Comment by udp | Hacker News Reader

Comment by udp | Hacker News Reader

It really doesn't matter. A string is a binary blob until you need to parse it, or display it. If you need to parse a string and it's UTF-8, you can pretend it's an ASCII string because the control characters you need for parsing are probably ASCII characters such as {, [, or ". UTF-8 continuation characters set a high bit to ensure that UTF-8 is a superset of ASCII, so you can use all of the standard C library functions. If you need to display a string, you're already using a font library which, in addition to providing the logic to iterate through the string character by character, can work out kerning etc.

I never understood the rationale for using encodings such as UTF-16. They seem to be the worst of both worlds: strings for which ASCII would be adequate take 2x the space, and the encoding is still multi-byte. I once worked with a Windows developer who swore blind that UTF-16 was not a multi-byte encoding. When I provided evidence to the contrary, they responded something along the lines of "ok, but who would ever need more than 16 bits worth of characters?". ¯\_(ツ)_/¯

The(/a) rationale for UTF-16 is that it's more space-efficient for languages that use high code points, like East Asian languages.

Sure, but the majority of the text computers have to parse has a syntax consisting of low codepoint ASCII characters (think XML tags vs. data). The fragments of text using a high codepoint language are usually only interesting at render time.

I'm not supporting the position; I'm just explaining what the argument for it is, since you said you never understood the rationale behind it and felt it gave you the worst of both worlds.

But I will say that the concern you have does not take into account the concerns other people have who make this argument.

The other aspect of importance is that languages with Chinese characters have higher information density per character, so they tend to even out in terms of actual storage requirements.

IIRC the real rationale is back in the early days people bet that 16 bits would be enough for a fixed length encoding, but the bet didn't pay off and now they're stuck with the worst of both worlds.

Depends what you mean by "rationale" I guess. I was trying to explain why one might still want to use it, not why one might be stuck using it.

Your explanation is correct. UTF-16 is a hack on top of a hack, and no one uses it in new software. However, Java and Javascript are stuck with it for legacy reasons.

It wouldn't be so bad if people just understood that there are (almost) no valid uses-cases for measuring the size of a string. As someone else mentioned, it's a binary blob for all intents and purposes, and if processing of its content needs to be done (such as displaying it on the screen, or performing, say, word-wrap) then it should be handed over to libraries that have been designed for this purpose, because these things are very complicated.

Simply by having a single non-ASCII character in my name, I'm seeing software fail on this on a regular basis.

East Asian HTML and XML documents are typically smaller in utf8 than utf16 because the of the markup. (on mobile so u can't prove it right now. maybe when I get home tonight) utf16 is a bad default for almost everybody.

The actual reason utf16 is used is because it was easier to port to utf16 from ucs16. Much the way it's often acceptable to assume utf8 is ASCII it's often acceptable to assume utf16 is utf8. Even today, I imagine there are a lot of applications which assume utf16 is actually ucs16.

That depends on what kind documents of course. It isn't usually true for text-centric content.

Here's Dazai Osamu's Hashire Melos: https://www.aozora.gr.jp/cards/000035/files/1567_14913.html. Despite having an unusually large amount of markup (most Japanese text wouldn't add the ruby characters), it's smaller as UTF-16 than UTF-8.

It took me a while to realize that by multi-byte you must mean variable-width. Of course UTF-16 is multi-byte, how could it not be? :)

Reply on news.ycombinator.com