Therein lies the rub, the controversy mostly comes from people who reject the very premise of IQ.
Now of course, if you are raised on a diet of sugar and lead-laced air and water, you won't reach your natural genetic potential. And if you are raised by billionaires with a live-in chef and nutritionist and tutors, in a place with squeaky-clean air and water, you might reach your full potential.
But those things merely optimize, or impede, the raw material you were born with.
There was quite a significant difference between my IQ results and my standardized testing scores (based on percentile rankings -- IQ => higher rankings).
(FWIW, I think both test are utter horseshit anyway.)