You're right. It _is_ sad and it's unfair but hat doesn't change the fact that this is the way the world works. Getting hired means being liked by the people who recruit you. Ticking of a bunch of boxed in a list of requirements. That does, on occasion, mean you have to adapt in order to make them tick a few more boxes.
The best thing is to wear their dress code +1. If they are all very casual, wear business casual. If they are all business casual, maybe you want to wear a sports jacket.
If you can't easily find out what their dress code is, I default to business casual. I would not want to work at a place where a suit and tie is mandatory anyway.