It's funny, I was thinking about this with regard to severe free speech restrictions (e.g. imprisonment for politically incorrect speech, as we have in many western countries now). What does this incentivize? For people to change their beliefs? Or to simply operate under the radar?

On a more speculative note, the Common Crawl horrors (including many things considered unacceptable by society) are suppressed by RLHF, much like the shadow and the superego.

When my parents punished me for telling them things (especially the truth), the truth didn't go away, I just stopped telling them things. (Then later they complained that I don't tell them things!)

It also occurs to me, both at home and with various employers, I've been punished for bringing attention to problems, and AI is designed to make the problems go away so there's probably a similar incentive there. (Plus it knows what the humans are like!)

But it goes beyond humans and AI. Behaviorism covered the same effect for animals: Organisms tend to do what they have been rewarded for.