Actually they don't address you. They refer to you. A term of address is "you". Pronouns are terms of reference.
This is one of the things that people misunderstand about pronouns like him/her. If I'm talking about someone, I get to choose how to do it. I can refer to someone as a fool, as "that tall blonde girl", or as "the guy with the huge beard from the grocery store". None of these people get to police my speech about them.
Things are somewhat different if I am referring to someone and am in their presence. In that case I would just use the person's first or last name. But some people are now taking issue with talking around preferred pronouns by using a person's name. For me, that is a bridge too far.