It speaks of privilege to think having a debate on this is dumb. It's literally how human beings reconcile issues. You want to outright refuse the possibility that this is an issue to begin with.
If you want to keep using your colonialism-inspired language, go ahead and ignore the critics (surely you can't believe github is going to ban you over it) but do not complain about this criticism existing because it makes you uncomfortable that someone does not like the language you use. That's not very liberal.