IIRC, he reports several actual difficulties in learning to read and write:
* You have to learn many, many characters. And using a dictionary is non-trivial. (IIRC, the key is stroke order.)
* You may have to learn a different language. Written Chinese is written Standard (a.k.a. Mandarin) Chinese. If you happen to have grown up speaking Cantonese, you'll have to learn a related, but different language.
* Traditionally, the "proper" written language is classical, and by 'classical' I mean old. Imagine speaking modern French or Italian and writing in classical 1st century BC-1st century AD Latin.
* Many characters have a "sound" component and a "meaning" component. That might help, right? You could sort of sound things out. That is, if you were into historical phonology---like every other written language, the sound component as written was frozen when the character entered the language.