And they aren't a very rich country now, after they lost all their colonies.
I don't wish to be harsh here, but how much of west's wealth is their own hard earned through their sweat is a question to ask.
There are also a lot of wealthy European countries that had a minimal presence abroad in the 19th century. There doesn't appear to any correlation between the modern wealth of western countries and their past colonialist ambitions.
It took me a bit to realise that you did answer the GP's "point".
(Some might argue that America is a de facto empire today, but even then most of its wealth is due to trade).