Maybe people good at math are actually great at people skills and you're just repeating a bias spread by a banality driven media.
There is a limit in the skills you can acquire. If you spend time and energy on flirting, learning to dress up, building social network, you will have less time to learn maths.
Because of course you'll have sport, music, games, books, movies, family and other study topics taking time as well.
Now maths incline kids, given the choice, will usually choose working on math than going to 10 groups of people to chit chat about superficial topics just to stay in the network.
Nothing wrong about it, just an observation.
I met some people that are both good at maths and with people skills. They are, however, adults. They started as nerds, and worked their people skills on the way up.
But starting with both, I never met one.
Because I was a geek, I was mostly surrounded by geeks. It gave me a huge and diverse sample to observe. Granted, it's not something you can consider scientific, since I'm the common ground and so the bias in all those observations.