Hacker News Reader

Comment by Jensson | Hacker News Reader

> The claim that LLMs are nothing more than fancy expensive markov chains is a dangerous oversimplification or, in other words, wrong.

But they are fancy expensive Markov chains. The learning here is that fancy expensive Markov chains can do a lot, not that the statement is wrong.

View on HN

They are fancy markov chains in the same way that your computer is a finite state machine. It is technically true, but it is an extremely unhelpful way of thinking about them.

Unhelpful is very different from wrong.

Just as simplified does not mean reduced. It can align and help understanding, even if incomplete.

Correct is very different from helpful

Just as munged does not mean increased. It can contextualize understanding, even if incomplete

Tell that to Alan Turing.

Alan Turing wouldn't think of a modern computer as a finite state machine either, he would think of it as a very good approximation to a turing machine.

Which is a finite state machine attached to an infinite tape.

Until it is helpful when you need to control the computer to process a sequence of data in a finite way similar to say a regular expression… #thisisfun

"All models are wrong, some are useful". You could also look at them as a highly compressed PGM representing every possible conversation you could have in every language the model knows that has been trimmed to only the highly probable paths through training. In that view word prediction is really navigating nodes and this would explain their ability to enforce global constraints as well as they do :) There isn't going to be one "correct" way to view them and different perspectives might offer different insights.

Not quite, I didn’t go into the details, but the analogy is flawed in a few ways: https://taoofmac.com/space/blog/2023/03/18/0140

They really aren't lol

Look up the definition for Markov chain.

A Markov chain satisfies the Markov property [1] in which the probability of an event is conditioned only on the previous event. Regardless of whether an LLM event is a character or a word, an LLM certainly does not follow a Markov process.

[1]: https://en.wikipedia.org/wiki/Markov_property

If you consider the entire input for the language model as the state, and consider the output to be the input concatenated with the next token, then it's a markov chain.

But that's only if you don't use BEAM search, which LLMs do use. If you use BEAM search with beam=4 (typical for LLMs), for example, then it does a tree search where it keeps track of the top 4 highest probability outputs, then I guess it's not a markov chain process anymore but it just uses a markov chain.

The state space of a Markov process must be fixed but is otherwise unspecified, which the LLM satisfies, so you are technically incorrect.

Reply on news.ycombinator.com