One reason for the "casualization" of dress were the world wars. Rationing and general mood (and having your country ripped apart) made it kinda impossible to dress extragantantly as a rule anymore. After the war ended, it kinda stuck (just like the daily shaving thing).