The Manifest Destiny doers didn't know it, but most of the Americas had been cleared a hundred years earlier by disease. Some estimate 95% fatality rates.
Is like you are trying to say, hey, I only killed 4%, the other 95% were already dead so is not so bad.