It is. And the reason for that is* surprising: the base knowledge corpus is clickbait internet articles.
*not at all
It is more surprising than that: it's because the humans in the loop like it.
The LLMs are going to turn us into stochastic parrots.
If you're being given an A-B choice between cinnamon-flavored shit and lutfisk-flavored shit, you might convince the experimenter that cinnamon-flavored shit ranks highly in user preferences.
I would expect a preference for lutefisk flavored shit, the problem is then you then deduce from that that people prefer lutefisk flavor over cinnamon, rather than that you should stop feeding people shit (and also stop feeding them lutefisk.)
I suspect it’s more like Claude etc. optimizing for a tone that gets the green checkmark from RLHF reviewers, probably over and over in many contexts where style is otherwise an incidental concern.