It's to the eternal shame of Poland, Germany, and many other countries in Europe, that this is not what they did after the war. It's frankly shameful that this is rarely enforced anywhere.
If you do 'right of return' without the implicit return of property thing, it's pretty hollow if you've actually had your house/farm stolen, but it also becomes just a way of having an immigration policy that discriminates on an ethnic or religious basis.
I feel like people are generally pretty fine with accepting colonial borders: basically the whole world outside europe is divided by them. As long as they have a chance to live in peace, build businesses, live normal lives, I don't think people get that exercised about these things, on the whole. If you look at the lives of Palestinians, they look pretty humiliating. If I lived that life, had no chance to build anything or hold onto anything, I expect I would be pretty angry about it.