I think it’s mainly that the UK, as the winner, got to write the history books.
(I’m not claiming that it cannot be partly racism — how many of “us” know about the Belgian Congo? — but mainly it’s “we are settlers, you are colonisers, they are invaders” and “our boys are heroes, your boys are soldiers, their boys are barbarians” in people’s minds).