You make some good points, and I probably should have made mentioned in my original post about how I too prefer wearing comfortable clothes when appropriate (I don't sit around my house in a suit like Barney Stinson). For example, I'm currently lounging around my kitchen in a t shirt and gym shorts. But when I need to be around people, especially at work, school, and social events, I personally feel more confident when my clothes fit better, are free from holes, and I'm wearing my nice shoes.
Also, for example, the other day I organized a "Restore the Fourth" rally for my home town. As the organizer, I felt it necessary to look nice and presentable because people generally respect nicer-dressed people, and I knew I would be interviewed for the evening news. If I had worn sneakers, gym shorts, an ironic t shirt, and a neck beard, things would not have run nearly as smoothly as they did.