Are you trying to suggest that this is a dichotomy? It seems to me that the common-sense position is (used to be?) that neither the person performing an action nor the person at the receiving end can be considered impartial and trusted to make that call.
Either way, the idea that "being hurt" is a state that the individual has full discretion to declare themselves as being in seems incompatible with our existing widespread moral intuition that A hurting B results in a moral and potentially legal claim to redress from B towards A; otherwise, anyone could trivially claim constantly being hurt by everyone and therefore entitle themselves to an arbitrary amount of restitution. One of the two therefore has to go for society to function: either you don't get to make the call if you are hurt, or it's a priori morally neutral to hurt you.