You describe it as something very cognitive, which is great, I say: something amenable to reason, and not compulsive.
But the other way it's used is similar to the recent (± 10 years) internet fad for diagnosing everybody you don't like as a "psychopath" or a "sociopath". Telling somebody that they lack empathy is like telling them they're a malfunctioning alien, and need to be in therapy and possibly manacles. For those who use "empathy" this way, we supposedly function through a natural, mysterious, almost supernatural and mystical instinct to care about those people and things who we should, morally, care about. And this instinct, which bypasses cognition to arrive directly as a feeling, holds society together, and anybody who doesn't have (or doesn't display, per your interesting point about how one can "still be an asshole") the correct feelings in the correct situations is suffering from a dangerous brain-wrong. Red flag!
This co-opts morality, and takes away your right to intervene cognitively about who or what you care about. But the person who supposedly lacks empathy is not really brain damaged. The person merely has different values, and doesn't care about the person or thing that you say they have to. Fictional instincts (or real but suppressed ones), and the fallacy that natural = good, should not be used as a lever to dictate other people's morality to them.
On the other hand, we totally should be considering other people's values, and enquiring about what those values are. Which may prevent much unnecessary fractiousness. Feel free to accuse people of failing to do that.