Hacker News Reader

Comment by xanderlewis | Hacker News Reader

> ...and then repeat until the model says stop.

How does this actually work? At a high level, the autoregressive token generation process makes sense to me, but how does it know when a sensible time to stop is rather than just going on forever or abruptly stopping after n tokens? Is it trained on text that has special 'stop' tokens inserted at the end of paragraphs, etc. and when it chucks out one of these the model halts?

View on HN

One of the tokens represents stopping. If you sample stop from the probability distribution instead of a normal text token, then you stop autoregressive sampling.

Open source models have an explicit endoftext token, yes.

You can set llama.cpp to keep generating after that and it often instantly goes offtopic.

Not only off topic but out of sample.

The old style LLM hallucinations where it starts spitting out random tokens like:

    nxrt$$.full #`,,, jump #43one daisy daisy give me your answer ###

Reply on news.ycombinator.com