Empathy isn't a sense, it's derivative from the senses and works by analogy with one's self. When what you sense from others seems similar to what you imagine others sense from you, by analogy you imagine that they have a similar inner life to you. The more difference from how you imagine that others would sense you, the more different you imagine their inner life to be from yours. When you see them to be similar to you, but more perfect in your opinion, you imagine that their inner life is more perfected than yours. If you see them as less contrived; less put together, you see their thought process as more honest, or more grounded than yours. If you see them as dirty or sloppy, you imagine their thoughts to be more perverse or more slipshod. If their movements or speech are slower than yours, you imagine them to be more stupid or more careful than you are.
It's the search for souls in other people, required to decide whether they feel pain in the same way that you do. You can never confirm that other people have souls and feel pain, it's a leap of faith that you have to make - what Hobbes would call a covenant - a unilateral agreement that you make with others without their participation in the hope that your good faith will lead them to reciprocate.
It's best if we just assume that everyone else is just like ourselves, and reserve our distrust for people that try to tell us otherwise.