It's about signaling. Sure, some people are able to overcome those filters through outstanding performance elsewhere, but why not give yourself every advantage possible? Why not strive to be the best version of yourself?
In my case, I design things for a living. What does it say about me if I didn't care about my personal appearance? "I care deeply about how things work and look. Yeah, but I couldn't be bothered to put on pants or a clean shirt today"
But to your point, I do agree with you. Do what's comfortable for you. In my case, dressing better (something I've tried to do for the past year or so, after several years of hoodies and ill-fitting jeans) has made me a better, more confident person.