Remember that heritability is a measure of a population, not an individual. So you're measuring the amount that genes vs environment contributes to a trait within that population. It's not an absolute number.
Edit: a good example of this in action is handedness, because different societies historically have different advantages to using different hands
> Second, the samples used to estimate the handedness heritability were from different places, different years or different centuries and were not always representative of the general population. Heritability is not an intrinsic property of a trait: it varies with phenotypic and additive genetic variances and thus can vary across populations and across generations