It's amusing and ironic that LLMs critique humans for copying their opinions from the crowd, while having way less diversity of answers to that question than humans would. More seriously I wonder what's going on here and would like to read some proper analysis. It's not the first time models from totally different labs, trained on separately gathered datasets, somehow converge on bizarrely similar answers to open questions. It's like another version of Elias and Elara The Lighthouse Keepers where every model wants to tell the same story with the same characters in them. The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping, different user feedbacks and conversations... why do they converge on nearly identical stories and opinions?
Why is this ironic? The actual number of different LLMs is tiny. And a lot of those are just distilled from SOTA models.
>The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping,
I think the opposite, the models actually converge over time. They ingest FAR more data than any human ever could in countless lifetimes. Any one particular bit of data may only have a part in a trillion of effect. From this data you start to reach means and statistical averages across all models.
Now, I don't always believe it will be this way. As training models get cheaper and faster we will see unique LLMs start to diverge and be used more often.
I don't think we're seeing averages or obvious fix points in the dataset. People went looking for the origins of Elias the Lighthouse Keeper and couldn't find one. Nobody predicted this outcome, it was discovered. It's a weirdly specific character and name choice, one that isn't obviously dictated by "the training set" (whatever that is), and yet they all reached the same place. How can such an obscure point in latent space be optimal for every model??
While I don't have any evidence based answers here, this could be some interesting mathematical artifact of the set of human knowledge and platonic algorithms.
Kind of like how carcinization turns some animals into similar shapes and forms, maybe there are just weird algorithmic shapes that emerge from how we use language. Of course we'd need to see more of these in different languages other than English to see if they disappear in some languages or appear more often in others.
This is just another one of those things about LLMs that we have no idea why it happens, and there are so many others already. And when you look at the pace things are progressing it seems we're building up new questions like this far faster than we are getting answers for them.
I think that this is most likely due to RLHF, as there's only a much smaller number of vendors here, and the likelihood is that the workers of the vendors preferentially prefer some kinds of actions.
I saw something like this in a previous life, where we had a small pool of human raters in cheaper places, which, when used as training data biased the models in a bunch of unexpected ways.
• They claimed this in a press release, not the actual paper they published. Suggests lack of confidence in the claim.
• Elias only appeared a hundred odd times in a million conversations. They have no viable explanation for why such a tiny amount of data would cause a totally skewed distribution of outputs, other than "it's a safe story" and "trying to avoid copyrighted characters", which is hardly an explanation at all?
• It's Cornell claiming that Cornell's work was important. Obvious risk of motivated reasoning.
• It implicitly assumes every major lab is distilling from ChatGPT without saying so. That's a big leap, given how much OpenAI and Anthropic hate distillation. Is it really plausible that Anthropic and Google are effectively distilling ChatGPT without realizing/admitting it?
• It seems to invert causality. WildChat came from an LLM, so if there's something latent in LLMs that makes Elias appear, you'd expect to see it in that dataset. Why do they assume that dataset caused the problem rather than merely show it? The obvious answer would be because the stories were told to the LLM by humans, but they don't claim that. They only say the character appeared in the dataset.
I would have been tempted to argue something like this: maybe OpenAI accidentally overfitted RLHF such that, like, one Kenyan contractor wrote a story they invented and they just happened to pick Elias and Elara, then the intermediate DNN fixated on that because of a lack of RLHF samples for fiction and that propagated through every model trained since because they don't randomly initialize the weights, but start each new pretraining run with the old weights adapted. So one guy doing RLHF work in the early days somehow propagated this story.
But that can't explain why every lab ended up in the same place, why models all converge on the same ideas - even image models. So... back to the drawing board. There's something weird about high dimensional space that is unexplained.
Wow, I hadn't heard of this. Fascinating to think about parallels with humans who also seem to develop similar proto-stories in independent populations.(e.g. religions involving Invisible gods).
To be fair, it's much (much!) easier to explain away an invisible god that nobody's ever seen, than a visible god that nobody's ever seen.
> There, within WildChat’s data, Hamilton and Mimno found a small sample of conversations that included Elias and familiar tales of lighthouses.
> “Based on the timeline, we’re fairly certain WildChat is an original source of this phenomenon,” Hamilton said, adding that future work will explore why LLMs prefer these lighthouse stories.
https://news.cornell.edu/stories/2026/10/who-elias-lighthous...
So, probably at least as much a quirk of the training data as a deep distillation of LLM psyche. Still fascinating though, especially IMO the broader pattern of tropes.
This is also why I think the 'acceleration' hype is probably unjustified. What seems vastly more likely is that we start to reach an asymptotic 'perfection' where models can accurately and consistently provide meaningful next token prediction with minimal hallucinations. And then a race downward to compact this into the most efficient package possible.
This would mean the models still would not be able to create things that do not follow in a reasonable probabilistic fashion given a knowledge of [everything], and so that is the domain, 'applied creativity', where I'd expect humans to remain dominant.
But in many sciences - esp purely theoretical ones - we have infinite datasets that we can generate and there can be no ultimate convergence.
Isn't that to be somewhat expected due to https://en.wikipedia.org/wiki/Aumann%27s_agreement_theorem ?
They’re trained by humans and human behavior fits a pattern. Models don’t have a mind of their own.
The same reason they do when it comes to coding?
Take a big enough sample size of human opinions and there seems to be some convergence in the statistical ether.
Like, if we polled 100000 humans, what answer would we converge to?
You assert that humans follow convenient beliefs therefore everyone in a group follows the same convenient beliefs therefore everyone belongs.
Convenient doesn't really mean anything in this context. What is the other option? Humans tend to make choices that are counterproductive/painful for themselves? Obviously we don't...
Ergo astra's response is extremely vapid. Low marks from me.
Anyway, I think I've pretty clearly described a positive feedback cycle in a way that distinguishes it from a tautology. I don't know what else to tell you on that point.
Convenience will kill humans eventually, that's what I always said.