As a person who is not particular agreeable to look at, and who can be quite introverted (maybe as a consequence), it is very noticeable that to me that "high-empathy" people are judging me on first impressions, not on my actions.
I don't think I'm a jerk. I always try to be extra nice (mostly because I want to, but also because I have to go the extra mile since I am otherwise socially challenged). Yet I just make others uncomfortable, in particular extroverts and socialites.