>Is GPT simply a Markov chain?
Yes, but much more efficiently stored and looked up than just a massive table. The way its very large space is encoded in a small amount of memory is what makes GPT so interesting.
Yes, but much more efficiently stored and looked up than just a massive table. The way its very large space is encoded in a small amount of memory is what makes GPT so interesting.
But given enough time one could model it as one.
Alternatively you could ask a neural net for the next state. Pass it the current token or the last 4000 token.
[1] https://towardsdatascience.com/text-generation-with-markov-c... [2] https://crowintelligence.org/2021/05/27/a-simple-markov-chai...