Hong Kong's population is only 7 million or so, a tiny fraction of 120 million travelers, assuming every Hongkonger went on international tours.
But perhaps you've never experienced poverty, so you may not know.
https://www.bloomberg.com/news/features/2022-11-02/hong-kong...
1.6 million out of 7.3 million people is definitely serious, but hardly a majority. I've been to the SAR a few times, and don't think it is especially poor, especially when compared to even China's richest city like Shanghai.
As for my own personal experiences, please don't make assumptions like that. You have no idea what I've been through in my life, and it is rather not relevant to this discussion.
When you add these to the people who are living in poverty, that doesn't leave for a lot of travelers.