The way I think about "intelligence" is not to mull over what it "really means" but to think about lots of common "intellectual" tasks (in which aptitude can be measured) that people are interested in: reading, writing, math, programming, solving puzzles, etc. Or ideally, theoretical "potential" in each of these tasks, since one person may put more or less effort into another field.
What's supported by research and anectodal evidence is that if you did principal components analysis on these measured "aptitudes", there would be a principal component explaining a lot of the variation. A good test can try to measure this principal component.
But when you're talking about the very highest levels of specific tasks like cycling or swimming, it would be reasonable to expect that amongst the population of already "elite athletes" there would be less of a dominant first principal component