Obviously, saying that creates a degree of investment in this other person that isn't required. But it's a middle ground. Everyone seems to think there are two options: peace out completely (zero investment), or do business without saying anything (100% investment). If you just point out the behavior, you put the ball in their court. It's scary because you risk pissing them off, but really, what does that matter? You already made up your mind about not doing business with them.
There are two likely responses to such a judgment: the angry/confused/awkward denial ("What the hell are you talking about? I'm a precious flower and everyone loves me because they are so blessed by my presence! Now run away, and never return!") or reflection ("Really? What made you think so?"). If you get #1, well, you're no worse off than when you started, except some douchebag with a lot of money doesn't like you. But if you get #2, you have a real opportunity to nudge someone in a more compassionate and healthy direction.
It's up to each individual person, of course, but I think the potential opportunity is worth the risk.