Hacker News Reader

Comment by skybrian | Hacker News Reader

Large language models work by choosing words [1] one at a time, much like autocomplete. It’s choosing the most consistent next word (with some randomness). It doesn’t actually care who wrote the previous words; whatever came before is the prompt.

When writing multi-word answers, it’s effectively prompting itself for another word. It’s going to try to choose a word that’s consistent with what it previously wrote. If it’s in the middle of writing a wrong answer, the best autocompletion is usually to continue to be wrong.

There will be changes in direction sometimes, but that’s because they also appear sometimes in the training text.

This explains why writing out the steps before the answer often works. It’s going to try to choose the words in the answer so they are consistent with the previous steps.

[1] tokens actually, but I’ll stick with “words” to keep things simple.

View on HN

> There will be changes in direction sometimes, but that’s because they also appear sometimes in the training text.

Yeah. My point is essentially that I think they don't appear in the training text often enough, since it's trained on written, edited text, which don't often contain changes of opinion mid-answer, unlike spoken text, for example.

Perhaps if it could somehow be trained on a dataset of humans writing responses, including editing (e.g. removal of previously-written words), that might lead to better responses, since it can then edit the answer after reasoning it through (in OP's example).

Or, to simplify things, perhaps it should be allowed to emit a token meaning "actually, scrap all that, let me start over" when it detects a contradiction in its own response so far. (The model would still continue to see its own response including that token, but the UI could remove everything until the last instance of the token, if any.)

Or, again alternatively, if we want a response closer to spoken text, it could be trained on dialogues from humans that change their mind occasionally, and then verbalize that explicitly (e.g. actually emit "actually, scrap all that, let me start over").

Except, why would it start over? A language model doesn’t actually have an opinion independent of whatever text it’s outputting. The prompt is the opinion. Start over with a new chat session and it will emit different words, causing it to have a different opinion. If you could edit the chat history directly, you would change its opinion and it would continue from there.

It’s true that if the training data contained a lot of text where the authors changed their minds a lot then the language model would learn to “change its mind” a lot in a similar way, but neither the old opinion nor the new opinion is its “real” opinion. It’s just the opinion it’s pretending to have now.

It would be tricky to come up with training data where, when an author changes its mind, the new text is always better than what they had before. Sometimes edits make things worse. It might be fun to try, though.

In the example, it "gained" additional information by reasoning through the steps. At that point, it should be more likely to give the correct answer.

In general, I think also when humans are able to reason about their opinion and change their mind if necessary, it leads to better results on average :)

That would be great. I don’t think it can detect the contradiction, though.

It doesn’t have any logical model behind it.

It essentially did detect a contradiction, and "resolved" it by trying to square its initial (wrong) response with the reasoning afterwards (which contained the correct response). Perhaps if it would've been able to delete the initial paragraph of its own response it might have.

Reply on news.ycombinator.com