ASCII and UTF-8 are too US centric. That's why adoption in places like China is so low.
Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?
Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?
I'd love to hear any specific counter-arguments.
More or less, I'd say that storage size of text usually doesn't matter.
Most of his points are not related to encoding size but to simplicity and standardization. I find those reasons to be very compelling.
I'm asking for clear counterarguments because I concede that my ASCII background could predispose me to UTF-8. Please be a little clearer, I really do want to know.
Does UTF-16 really have such a great advantage for non-Roman writing systems? Or is this motivated more by a disliking for Anglocentrism?
Yes, ASCII is obviously too US centric; you can't encode any writing systems other than the Roman alphabet in ASCII. However, that's not at question here. The question is, which Unicode encoding should you use, so you can represent all writing systems in a single encoding. And the major contenders are UTF-8 and UTF-16. The point of this article is, for that purpose, UTF-8 is a far better choice.
> Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?
What do you mean by a "proper way"? If size is what you care about, just compress your data. Compression will do a lot better for a much wider range of data than some clever encoding will. UTF-8 is a carefully constructed encoding designed to meet several design criteria. For instance, you could get better size for a wider range of character sets by having a single byte to represent switching between character sets; so you could use that byte, and then a whole bunch of 2 byte CJK characters. But that would defeat one of the design goals of UTF-8, which is to be self synchronizing. That means that if you get a partial sequence (such as a sequence that has been truncated), you can start decoding the characters after a fixed number of bytes. In the case of UTF-8, you will never have to go more than 3 bytes before you can start decoding again. In my hypothetical scheme where certain symbols were used to switch between character sets, you would not be able to interpret anything until you found the next such symbol. This makes UTF-8 more robust in the face of errors.
Another design goal of UTF-8 was to be backwards-compatible with ASCII. Like it or not, ASCII has been the standard encoding for decades, and there is a lot of text in ASCII and a lot of software that uses ASCII delimiters and the like.
So, while it would be possible, in theory, to define a character encoding that is more "fair" than UTF-8, that ignores many of the other goals of the design of UTF-8. And UTF-8 is widely supported and used (it is the most popular encoding on the Web, even in places like Japan, and a close second in China), while a new encoding would require another large, global, and painful transition process to introduce.
For instance: EUC-JP in Japan, BIG5 in Taiwan, GB in China, etc. Different EUC encodings are variable-length and are a lot more efficient since they put common subsets in lower parts of the table for each language, close to each other so they also compress better, while allowing tricks for text matching and searches (not really necessary for web sites, but it's nice to use the same encoding throughout applications sometimes). Russian and Greek are also basically multiplied by two in size.
There are a lot of other considerations.
If you think a 30%+ saving in size (and latency) is not a big deal, then you're a lot more likely to lose to local competitors. Note that gzipped or otherwise compressed text makes differences even worse at least in the case of Japanese - where UTF-8 text gets de-aligned all over the place to odd byte sizes and compresses worse. Add to that the fact that Asians browse the net A LOT from the phone and have done so for much longer than westerners and in a bigger percentage, and you have your problem exacerbated even further.
There is a lot more to consider and like it or not it's not as simple as "UTF-8 for everything and everybody, ever!"
In the global Internet, UTF-16 and UTF-8 are the only games in town.
Then UTF (and EUC's) are the way to go.
It's not like you have to use the same encoding all the time.
Then you are going to feed someone garbage. Why feed people garbage?
Not if you know what you're doing. Not any more than using utf8 exclusively all the time and for all purposes.
This is nice in theory. In practice, people make mistakes. Make it easy on yourself.
> Not any more than using utf8 exclusively all the time and for all purposes.
Maybe I was unclear: Feeding me Chinese text in UTF-8 is not garbage. Feeding me anything in one of the GB encodings is garbage.
Garbage, to me, is text in an encoding I can't handle. If you only use UTF-8, that cannot possibly happen.
That was not what this article measured. UTF-8 and UTF-16 compressed to virtually the same size.
That's precisely what I meant. Simple variable-length compression.
Whether you agree with the sentiment or not, pg's oldish comment on the issue at least establishes the existence of the behavior several years ago, and has been taken by many to be the final word on the acceptability of the practice: http://news.ycombinator.com/item?id=117171
(edit: spelling)