Because IQ is a very specific thing.

It doesn't mean you know a lot or are good at high school maths.

IIRC an ideal IQ test would be something you could take even if you can't read and haven't learned maths and it will tell you something about your general problem solving skills.

And as we all know, people can be excellent problem solvers without being very athletic at all, so mixing that into the score would actually make it less useful.