Is there a way to know how much to care? We do all sorts of things that displace species, so having some way of contextualizing and weighting the harm against the benefit would be helpful.
However, as with all human endeavours, scaling up will scale up the problem. And we won't pay attention to that until it's a huge issue.
If you want to see what a tourism place looks like when you destroy its ecology, you only need look at the abandoned areas around the Salton Sea.
Species will always come and go, and they're going to die off faster than we can protect them, sadly. We only get to decide which ones to prioritize, at different personal, local, national, global levels etc.
I'll try to find some analyses this weekend if I can.