The book "Exercised" by the Harvard anthropologist Daniel Lieberman mentions exactly this claim - that in hunter gatherer societies, people actually rest for most of the day. Still their average lifespan is about 70 years.
> we see that on average 57 percent, 64 percent, and 67 percent of children born survive to age 15 years among hunter-gatherers, forager-horticulturalists, and acculturated hunter-gatherers.
https://gurven.anth.ucsb.edu/sites/secure.lsit.ucsb.edu.anth...