Like you said, the distinction between meaning and information is largely superficial and arbitrary. Letters are symbols, symbols carry meaning - a particular squiggle only contains information because it has meaning (outside of models and other artificial systems). If I don't read Sanskrit, one letter contains zero information/meaning for me.

"If something is more probable, it contains less information." I don't understand, why would "ee" contain less information than "hh"?

I'm also wondering where his comparison of English to Japanese came from:

"For instance, we could compare the number of bits per character in Japanese versus English (it turns out that English is only 60% as compact as Japanese, which explains why Japanese book are always around two-thirds the size of their English counterparts...)."

Are we talking about kanji or hirigana/katakana? If we're talking about kanji, we're talking about thousands of pictographic symbols (a lot of which are multi-syllabic), which are fundamentally different in nature than English letters (as for the 60% more compact bit, citation please?). Based on his own assertion, "Let’s say that there are 32 letters and punctuation marks... each character represents 5 bits of information ... a total of 5*16 = 80 bits of information." There's somewhere between 2000-7000 kanji ... so each character would represent...?