If youre doomed by a fancy markov chain chatbot, you were never gonna make it in the first place.
almost like a fancy markov chain? :)
vs
> An attention mechanism allows the modelling of dependencies without regard for the distance in either input or output sequences.
See the difference? In Markov Chains it is enough to know the previous state, while in transformers you need all previous states. It would be a great thing if we could reduce dependency of all previous states, like an RNN, maybe RWKV will do it.
The attention layer just looks at that bounded state. GPT-3 for example looks at a few thousand tokens, those are its state, it is bounded so it doesn't look at all previous tokens.
> A second-order Markov chain can be introduced by considering the current state and also the previous state, as indicated in the second table.
i.e., a higher-order Markov chain can depend on several of the previous states.
So, if a certain transformer model accepts up to 20k tokens as input, it can certainly be seen as a 20000'th order Markov chain process (whether it is useful to do so or not can be debated, but not the fact that it can be seen as such, since it complies with the definition of a Markov chain).