He's describing the people he saw every day for decades. If you think you understand those people you've never met more than he did, you're the one who lacks an understanding of empathy.
If society wanted to be more humane and empathetic, it would force all young adults to be homeless and impoverished for some time, and working three jobs to get a taste for how it is for everyone else that isn't sheltered or privileged. The golden rule and empathy isn't practiced as much anymore because there are greater convenient distractions, selfishness, ignorance and lack of community that has the effect of atomizing people from each other... we are mostly divided-and-conquered and enslaved.