You make a good point but you're also wrong.
The way the colonies (and later United States) treated the natives was absolutely deplorable and immoral. The US (from the founding fathers until very recent times) is as guilty as you say.
HOWEVER... it cannot be denied that the accidental spread of European diseases wiped out over 50% of all natives before the era of colonization even began. Even today if contact is made with isolated peoples great care must be taken and medical services kept on stand-by. A disease ravaging the isolated group is a very, very real possibility.
It also means if humanity does every colonize another planet (even in our own solar system), if the flow of people ever stops we may not be able to re-establish it due to disease risk.