Hacker News Reader

Comment by StilesCrisis | Hacker News Reader

Has anyone ever tried giving LLMs a backspace? I have long wondered if their responses could be better if they had the ability to correct earlier parts of their text. Imagine how much harder it would be for us to write coherently with no backspace!

View on HN

Thinking mode is exactly this. The LLM is allowed to dump thoughts, correcting itself along the way, and only surfaces the cleaned-up final answer to the user

It's not exactly this. It's something like it. But it's not a token that erases a previous token. Not that I know how that would work or how you'd get training data (edit histories of internet comments seem too few).

(You could have a token that hides previous tokens, but that'd be rather closer to CoT.)

Backtracking, of sorts. The LLM descends a seemingly fruitful path, and all of a sudden that path seems less so. So you backtrack a few tokens ("erasing"/"undoing") and resample the probability distribution a bit back in the stream.

Would it retain some memory of the fact that it backtracked? That path becoming less likely somehow?

Yeah that's what I was imagining. No idea how though, so this is just rambling on my part :)

ChatGPT does this. It will present an initial response, then change its mind partway through the thinking process and present a different response.

Oh wow, I'd like to see this. Any prompt which reliably triggers it? (Seahorse emoji comes to mind!)

It does it quite consistently on higher reasoning.

I believe they have made display answer a tool call so that it can keep updating its answer as it thinks, instead of doing all the reasoning up front, essentially inverting the usual flow. You can use this pattern with any model that supports tool calling.

Yes, it was invented by an anonymous user on 4chan in the early 2020s, then sanitized by a Google paper some years later as CoT (chain of thought) or thinking tags, where the model iterates on its own before writing a final response.

All models do it now.

Reply on news.ycombinator.com