The economic answer would be that the possible costs it could impose on you can be expressed as violations of your property rights, and if they wished to do this then they'd need to compensate you, or else you could sue them for redress.
>I know, but it's an absurd stance, isn't it? It let's you do things like seeing a person that has a heart attack and thinking "I see you have a heart attack, but why should I help you, maybe it's your preference!"
Being absurd doesn't necessarily make it less philosophically valid. In one sense the whole concept of morality is absurd, via https://en.wikipedia.org/wiki/Regress_argument. There's even a school of philosophy called Absurdism that based around recognition of the absurdity of seeking absolute values that inherently cannot be found: https://en.wikipedia.org/wiki/Absurdism.
On a more concrete level, the rejection of quantifiable utility is based on rejection of the extra assumptions it requires. Two in particular: firstly, it assumes some kind of simple relationship, so that if person X buys 10 apples for $100, then they valued each apple at $10 (linear), or maybe the first at $50, the second at 25$, etc. (polynomial). There is however no irrefutable logical justification for the assumption, e.g. perhaps they valued the first at $30, the second at $1, the third at $4, the fourth at $5. Secondly, it assumes that a person's preferences at time t have any relation to their preferences at time t+1.