Yes, there are absolutely limits on your physical development that you are born with. You won't run 100m under 10s if you're a woman, and you're much more likely if you have recent roots in certain countries. You won't get to play professional basketball unless you're quite a lot taller than average. If you're going to win Mr Olympia, you need to possess the kind of body that can be turned into a hunk of muscle.
But you can do better than most people at almost everything by practicing. You can cut your 100m time just by losing the gut. You can practice throwing the ball in the hoop by yourself. You can lift weights and you'll get more muscular.
Intelligence is no different. If you're going to be a research mathematician, it's going to be incredibly tough competing with the smartest kids in your generation. But if you just want to appreciate quantum physics or complex numbers, it's actually quite low hanging fruit that you will figure out if you want to. Most people do not figure out complex numbers despite it being taught at every high school.
The thing with intelligence, and I suppose we mean the mathematical kind, is it's invisible until you interact with people. If you're very weak or fat there's reasons to do something about it. If you just don't know how vector calculus works, chances are you won't get outed.
Our culture in the West also has a static view of intelligence, the opposite of Dweck's growth mindset, which really harms everyone who should be encouraged to practice getting smarter. The loss of not stopping people without world class brains from pursuing math professorships is greatly outweighed by the loss of everyday people not improving strengthening their thinking skills.