Hacker News Reader

Comment by throwawayffffas | Hacker News Reader

Garbage in garbage out. That's how machine learning works it's not a "programming error", it's their data.

In the case of this and the earlier "white genocide" replies, it is way more likely someone changed the system prompts than that someone tampered with the training data, considering the conspiracy theory was brought up unprompted in wildly unrelated situations [1]

[1] As one example, https://bsky.app/profile/jdcmedlock.bsky.social/post/3lp6eal...

I wonder who that someone might be.

Oh my theory is not that someone tampered with the training data. It's that their data is sourced from bad sources think 4chan, 8chan, etc.

Obviously I can only speculate since I neither have access to their dataset nor interest in paying for API access, but crawling and dataset cleaning have gotten much better since the GPT-2 days, especially after Microsoft's PHI models [1] demonstrated how much dataset construction matters for parameter efficiency and toxicity. Having some basic content filtering is a pretty established part of data cleanup -- e.g. the fastText toxicity classifiers in the Dolma pipeline [2] -- which obviously still leaves in bad data, but certainly won't leave in the entirety of /b/

If shoddy data collection was the problem, we should expect the model to do much worse on overall leaderboards like [3], which require models to answer questions without sudden detours into Holocaust denialism. A change to the system prompt is more consistent with this, and as an added benefit, only requires one person to be completely out of their gourd.

[1] https://www.microsoft.com/en-us/research/publication/textboo...

[2] https://arxiv.org/pdf/2402.00159

[2] https://livebench.ai/#/

Reply on news.ycombinator.com