This is a common misconception about how life expectancy works. If you have 10 children, 9 of them die at age 5, and the 10th dies at age 55, the life expectancy of your children is 10. Those 9 children who die early aren't going to go through puberty and become parents, but the life expectancy of those who do reach that stage is decently long. Low life expectancy doesn't mean you're considered some decrepit geriatric at age 12; it means that you get to grow up with lots of siblings who didn't make it.

When you look at pre-modern societies (essentially until around 1900 or so), there are four large causes of low life expectancy that don't really exist anymore. Infant mortality was extremely high (a quarter or so of live births didn't make it more than a few days). Then child mortality itself was high (you had perhaps a 50-50 chance of making it to puberty). For young women, there's an extra hit to life expectancy due to risky childbirth (you have about a 10% lifetime chance of dying in childbirth). For young men, they get extra risk due to endemic warfare--the baseline casualty rate of premodern warfare is basically WW2-levels of total war. Of all these factors, the infant mortality is the heaviest drag on life expectancy, as dying at age 0 really pulls down the average lifespan.