Well of course they don’t actively worry about your hair, but most certainly it’s one of the factors on whether people find you attractive or not. And that can still have real consequences even if people aren’t “worried about you.”
And the issue of whether people generally care misses the point. A man wants her to find him attractive.
Beauty is well researched & documented to affect one's social standing, how much people trust you, career success, mate/family success, and many other factors.