My personal preference is to never wear a button down with a 'formal' suit. I wear suits for work every day (and I am trying to single-handedly get the 3-button suit to make a comeback) and will not use a button-down with them. There's no true right or wrong answer, but as has been more elegantly stated before this post, most people will equate the button down to a more casual look.