No, the recipient determines if it is harmful. Malice is characterized by intent.

Hard disagree. Even if it is hurtful for the recipient, it might have been in good taste and honest, playful banter.

If you let your behaviour be defined by the most sensitive and high σ recipients, you cease to be able to conduct any form of communication apart from talking about the weather. And that might even offend some people then.

malicious

adjective Having the nature of or resulting from malice; deliberately harmful; spiteful.

malice noun 1. A desire to harm others or to see others suffer; extreme ill will or spite. 2. The intent to commit an unlawful act without justification or excuse. 3. An improper motive for an action, such as desire to cause injury to another.