This is an excellent point that I'm amazed no one has made. It's entirely possible to dress very well but without wearing a suit/blazer/wool pants, etc. There are tasteful, well tailored, yet casual clothes all over the place.
A slob in a suit is still a slob.