Hacker News Reader

Comment by hcarvalhoalves | Hacker News Reader

A Markov chain generated something that doesn’t correspond to things that happen in the real world? How surprising!

We’ll get news like this for a few years until the hype dies off I guess.

ChatGPT is not a Markov chain, rather it's a large language model. On a very high level, it tries to predict the next word in a sentence such that it most closely resembles the human written speech it was trained on.

Markov chains, however, are higher order statistical models. For example, they can predict the average waiting time at a service point based on the current state of the waiting queue. As opposed to just averaging over a bunch of values, Markov chains base their predictions by applying stochastic models to higher orders (i.e. accounting for current and previous states), yielding potentially higher accuracy.

The base LLM is absolutely a Markov chain. It gives a probability distribution over the next token, given the previous 2048 or however many previous tokens in the sequence. You can't be more of a Markov chain than that.

It doesn’t sound like you understand what a Markov chain nor an LLM is, you’re just parroting something you heard and that stuck in your head. Please don’t?

I mean if you read the definitions you can see that it's a Markov chain. I don't know what else to tell you.

Maybe you have the idea that only low order Markov chains, like second or third order Markov chains are "true" Markov chains and that 2048th order Markov chains shouldn't be called Markov chains anymore?

Maybe you think that the transition matrix must be constructed explicitly as a square of numbers stored in the memory of the computer for it to count as a Markov chain? It's not true, you can have the transition matrix be implicit and it's still a Markov chain.

Maybe you think not just any transition matrix is allowed for a Markov chain, and that the ones for LLMs are (implicitly) defined in ways that disallow it from being a Markov chain? For example, maybe you think the transition probabilities are required to be constructed by counting how many times the next token followed the context in the training text. Well that wouldn't be true, you can for example make a transition matrix that gives nonzero probability to a token even if that one has never followed its context in the training data, and it would still be a Markov chain.

Maybe you think that if I'm defending someone who calls an LLM a Markov chain, then I'm also defending the "stochastic parrot" take and the takes of ones like Gary Marcus. Well that's not true. I can recognize the LLM is a gigantic Markov chain and I also think the "stochastic parrot" take isn't a good one. My favorite take is the simulators one https://generative.ink/posts/simulators/

Markov Chains are analytical models, as opposed to LLMs.

Big difference.

An LLM (one that for example gives a probability distribution over tokens that follow a 2048 token context sequence) is a giant high-order Markov chain with an implicit transition matrix. I guess you are hung up on the fact that the full transition matrix of an LLM is never actually constructed in memory in closed form. Maybe according to your definition that makes it not an 'analytical model'.

besides, where did you get the 2048 token number from?

The LLM GPT-3 for example has a 2048-token-long context.

Calling chatgpt a Markov chain is like calling a weather forecast linear regression.

Reply on news.ycombinator.com