I would be bothered that someone dressing excessively formal might act excessively formally - after all, dressing formally is normally a deliberate display of conformity.
Or, you know, some of us just don't like looking like schlubs all of the time. What's wrong with that?