There are all types of button-down shirt guys, from dressy to messy, and (at least) two who fit into the casual bucket are mr downtown and mr uptown.

Of course there is crossover.  Both are interested in style and like to buy premium.  For the shirt, it’s in the details and how it’s worn. 

mr uptown in lightweight yarn/dye cotton with mother of pearl buttons

mr downtown in soft cotton/linen blend on indigo base, heavily laundered

Let’s face it, both are cool dudes.