That particular stuff is actually pretty typical. I have a textbook that shows similar results on Shakespeare using N-grams from years ago.
With ngrams, Markov models are perfectly sufficient. With individual characters, complex concepts need to be remembered across many, many characters of input.