If the context window is finite, then LLMs actually are Markov chains. It's just that they're a much more efficient way of representing transition probabilities than storing them all in a giant lookup table.