A Markov chain satisfies the Markov property [1] in which the probability of an event is conditioned only on the previous event. Regardless of whether an LLM event is a character or a word, an LLM certainly does not follow a Markov process.
But that's only if you don't use BEAM search, which LLMs do use. If you use BEAM search with beam=4 (typical for LLMs), for example, then it does a tree search where it keeps track of the top 4 highest probability outputs, then I guess it's not a markov chain process anymore but it just uses a markov chain.