A few centuries ago, when you say people could have fun doing nothing at all, what you call "a great deal of suffering" in contemporary society is actually materially better than even the middle class lived back then. Much better. It's just that people are wired to compare themselves to their community, and by default that's what people base their self-worth and contentment on, unless they do some careful introspection and consciously change their source of sense of self-esteem.
A few centuries ago, the average family lived on a small farm with little education, no running water, no electricity, and little excess. A large fraction of the children would die before their first birthday, and dying of bacterial infection was quite likely at any age. The boys could expect nothing more than a life full of hard physical work on the farm, and the girls could expect to get married or, if they were very lucky, get a profession like nursing or teaching. Yet, this average family was happy, and it wasn't because of their material possessions or adequate healthcare (clearly). It was because they were grateful for what they had, and because of a supportive community who were just like them and were always there for them.
You're missing the point of the quote you pasted. His father had everything he ever wanted because he didn't value material wealth, and was perfectly content with what I imagine you (and probably me too) would call "suffering", because he had friends, family, hobbies, time, and relatively good health. That's really all we need.