It's the reason that if you ask a random child to draw a house, you'll end up with the classic “house with a pitched roof”: a square plus a triangular roof, one door, two windows, and sometimes a chimney.
If instead you ask to draw some fancy interestingly looking house you'll get more variety - now the task is not to have the question giver recognize the house in their drawing, but to show how they imagine fancy or interesting house.
Can you prompt around it? Naturally, but that's kind of my point. The more generic your prompt, the more generic the response.
In my experience, simply adding modifiers like "interesting, unusual, bizarre, imaginative" doesn't necessarily help either. Try asking an LLM for a "insert aforementioned qualifier adjective" fantasy short story in independent context/ conversations a few dozen times and you'll quickly see what I mean.
Yeah, I hear what you’re saying. I guess it depends on how we define “notably different.” As I mentioned elsewhere, if you ask a random group of 12 people to draw a house on a piece of paper, you’ll end up with almost identical results. Sure, there will be some minor differences but for the most part, it’s going to be the same square with a triangular roof, two windows, one door, and maybe a chimney.
> AFAIK, the answer is that it's RLHF that causes models to collapse to a specific style, because base models tend to score much better on creativity and output diversity evaluations than post-trained ones.
Interesting. I'd love to see a paper on this. I wouldn't mind trying some back-of-the-envelope tests on a RHLF vs Base version of a locally hostable Qwen model too!
The first time I personally heard of RLHF ruining some model qualities was actually in the GPT-4 paper (march 2023); figure 8 there shows that the base model has near-perfect calibration on multiple-choice questions (!) and this stops being true for the RLHFed model: https://arxiv.org/abs/2303.08774v6
I think this is the same sort of problem - that the miscalibration of RLHFed models also explains their lack of diversity. In the training dataset, for a prompt like "you're a helpful assistant, yadda yadda, write a site", there's a large range of possible completions which are quite different and yet similarly probable, and if a base model is well-calibrated on that space, that lets you sample from that entire space and get many diverse websites. But an RL-trained model is miscalibrated and chooses a much smaller subspace of completions to assign unnaturally high logprobs to, so no matter how you sample, all the websites you get will look the same. It's not obvious to me why this happens; I wouldn't be surprised if, for example, a new optimization algorithm gets discovered that prevents RLHF from having such effects.
As for studies that measure this effect, I just found these: https://arxiv.org/abs/2505.00047 , https://arxiv.org/abs/2507.20956 , https://arxiv.org/abs/2604.16027
Sort of a tangent, but if you remember AI Dungeon when it first came out, it was based on a fine-tuned GPT-2 checkpoint which of course was pre-RLHF. While its adherence to user prompts wasn’t fantastic, it got a lot of praise for its diversity at the time.
I almost wonder if you could do something like this: have a non-RLHF model that gets the user’s request and just spits out a cloud of possibilities. It might not be the most tightly tuned, but then you could have a RLHF model take that output and use it as a sort of quasi-seed to increase diversity, while still retaining the RLHF advantages of producing an ultimate response that aligns with the user’s initial request.
Looking at the cited-by I found a paper that evaluated a bunch of such methods by LLM-judged novelty, https://arxiv.org/abs/2606.12790 . CD is weirdly weak according to these results, they get better changes with other methods - but all of the others are less technical and basically amount to asking the model to try harder at being original. I wonder whether CD would work for a task like programming...
For most kids there isn't enough time to draw a better house - they want to go play. They also are well aware that there is only so much time in class and the better house wouldn't be done and in turn look even worse.