Similarly, climate change isn't a threat to life on earth as we know it, but it does inconvenience a few species. And asking people to move to another country because you drowned their island or burnt their city isn't a great way to make friends.
By and large the Earth is a big wet rock that suffers a variety of misfortunes from time to time, and it tends to suck for things living on it during those times. Climate change just happens to be one of those periods of unpleasantness, except this one is human caused and happens on a human timescale. If humans sent a asteroid hurting towards ourselves, to hit us in a few decades, you'd probably see a few protests and strongly worded science studies about that too.