Hong Kong is a city state however, so it is probably true that most of their residents travel abroad each year (if we count China as traveling abroad, but even if we don't, > 50% is probably about right).
But perhaps you've never experienced poverty, so you may not know.
https://www.bloomberg.com/news/features/2022-11-02/hong-kong...
1.6 million out of 7.3 million people is definitely serious, but hardly a majority. I've been to the SAR a few times, and don't think it is especially poor, especially when compared to even China's richest city like Shanghai.
As for my own personal experiences, please don't make assumptions like that. You have no idea what I've been through in my life, and it is rather not relevant to this discussion.
When you add these to the people who are living in poverty, that doesn't leave for a lot of travelers.