2nd, this seems very much to be predicated on Cartesian dualism, that the mind is independent of the body. Would someone growing up in a society where they were subjected to daily derogatory statements in regards to some group to which they belong, would their physical wellbeing be divorced from their mental wellbeing?
If the subjective is to be discounted, then there is little in the realm of human reality that will remain. Everything that a human perceives is part of their imagination. Our consciousness does not touch the real (so far as we may know); it constructs sensory data models that are by definition subjective.
Why should the speaker be granted carte blanche any more than the listener? The language game is not merely the pitcher and the passive receiver. Even silence is an interpretable phenomena. There no way out of the dance of meaning.