Hacker News Reader

Comment by simias | Hacker News Reader

>Even though this is 2018, occasionally someone will try to claim in conversation with me that UTF-16 is better than UTF-8.

I'm curious, what are their arguments? UTF-16 is pretty objectively bad at everything, it's not compatible with ascii and C-strings like UTF-8 and it's not a decent constant-width encoding for codepoints like UTF-32 (and even that is not necessarily useful anyway since you often care more about characters than codepoints). AFAIK the only advantage of UTF-16 is that it's a little more efficient when encoding text written in certain scripts.

View on HN

People who don't understand surrogate pairs and think that UTF-16 == UCS-2 tend to argue that UTF-16 is better because (in their minds) it's a fixed-sized character encoding (it's not).

Even UTF-32 is not a fixed-sized character encoding!! It's a fixed-sized codepoint encoding. Characters can be composed of more than one codepoint. Even if you think "hey, I'll use pre-composed codepoints", you'll fail because not every legitimate, canonically decomposed character has a single-codepoint pre-composition.

The disadvantages to UTF-16 far outweigh its one advantage (the one you mentioned).

(There is one more advantage to UTF-16, and it's that Win32 uses it all over the place. But that's not much of an advantage, and Windows does seem to be making progress towards putting UTF-8 on an equal footing (if not better).)

Does Win32 use UTF-16 or WTF-16?

For path and object names, it's the latter.

WTF-16 "standard" is just UTF-16 that allows invalid surrogate pairs. Windows kernel style handling.

It's a shame you were voted down for a perfectly legitimate (if misleading) question.

This is a very common gotcha most developers ignore until they encounter their first invalid surrogate pair file path... UTF-16 can't represent all possible Windows file names or paths.

Was it misleading? I was just trying to understand if Win32 used WTF-16 everywhere (and some strings mostly happened to be valid UTF-16), or if there was some place where it used UTF instead.

I think object names and paths are WTF-16, that's where NT kernel is involved. Note that Microsoft doesn't call it that way, but it's a useful naming distinction regardless.

Applications can of course use whatever they please. Just need to be careful with the possible invalid object or file name surrogate pairs, not to fail on those and to pass them with alteration.

Well, misleading in that sense you did get the downvotes... That said I find WTF-16 pretty appropriate name for what Windows does internally on the kernel level.

In general kernels know nothing about codesets, but filesystems do (e.g., ZFS definitely does). I call this just-use-16 and just-use-8 depending on whether the strings are arrays of bytes or of uint16_t's.

I'll bet they saw it and thought you were just being insulting (what-the-fuck-16), but it apparently stands for wobbly-transformation-format. Had to look this up myself to be sure.

This is a historical oddity. At the time Apple and Microsoft (plus a few others) game up with Unicode the assumption was that we’d be able to fit everything into 16 bits.

That turned out not to be true but binary compatibility made switching to UTF-8 impossible.

Not just binary compatibility, also API compatibility. All of the performance guarantees of NSString are based around the internal representation being UTF-16 (or ASCII). Similarly -[NSString length] is defined as the number of UTF-16 code units, and all of the subscripting / range operations are defined in terms of UTF-16 code units.

Same thing goes for Java and .NET.

It's somewhat ironic that those who were the earliest on the Unicode bandwagon ended up saddled with all this compatibility garbage...

What gets me is that this affects later languages too. For example, Swift's internal String storage is either ASCII or UTF-16, precisely because it needs to maintain NSString's performance guarantees when bridging to Obj-C.

That said, it seems the plan for Swift 5 is to change String's internal storage to be UTF-8, and then when bridging to NSString it will provide amortized constant-time UTF-16 codepoint access by calculating and caching "breadcrumbs" for fast lookup of UTF-16 offsets.

Plus while UTF-32 is currently defined as equivalent UCS-4, the surrogate pairs tricks of UTF-8/UTF-16 are still theoretically possible even if they don't currently encode anything. Not that we have a reason today to expect there to be a plane bigger than the Astral Plane, but in the 90s no one expected the Astral Plane to open up. Anyone picking UCS-4 today and pretending its UTF-32 but not prepared to handle surrogate pairs for the Currently Imaginary Plane beyond, is potentially making the exact same mistakes of UCS-2 early adopters and will have the same UTF-16/WTF-16 headaches should that day arrive.

I expect UTF-16 to die eventually and let us have more than 21 bits of codepoints, but we'll never ever need even 30 bits, let alone 32 bits, of codepoints.

I have to imagine that you're just jesting :0

It's not entirely a jest, but it is funny to think about. The question will be then, as it has been every time before: what do we mean by "Universal" in Unicode?

One very obvious jest is that at the current rate of emoji expansion it may be inevitable to open up the next plane just for emoji.

Less of a jest is that there's always more symbols to encode. We've been a wonderfully creative species, and we're still rediscovering old written languages and discovering new bits of physics and math that can benefit from new symbols. We've got tons of interesting fictional language encodings to consider that are currently not likely to be encoded in Unicode today because they are locked in that struggle between the needs of Academia, the interests the Enthusiasts, and the money/complications of Intellectual Property laws. (Klingon, Tolkien's Elvish languages, D'ni, Timelord Sigils, etc.) But if those seesaws tip towards enthusiasts or academia that changes quickly. Meanwhile, creative artists keep inventing new glyphs and languages to encode.

That's just our one species on this planet. What if we discover more or get better at encoding existing non-human languages? Then we really get to test that "Universal" bit in the name, huh? (That's also obviously a bit of jest, but an interesting sci-fi hope/yearning there, too. It would be nice to encode some non-human languages for a change.)

So, all of the above, based on current assumptions (emojis will slow down, IP law remains vicious, Fermi's Paradox won't been solved in our lifetimes, etc) may still be unlikely to push us past the Astral Plane, but that doesn't make it silly to assume that we may never need to open the next plane.

There's always more glyphs to encode, but I think it will be a long time before we reach hundreds of millions of glyphs, even if we assigned codepoints to every fictional script and every glyph used in dead scripts, and every variation used in medieval and ancient times, and every future new emoticon or glyph for new scripts or fashionable things.

Remember, you need a fairly large number of users for each glyph, otherwise there's no point. We're not going to have per-person glyphs, as no one could understand enough of them for them to be useful. There aren't going to be too many new scripts with users in just the hundreds of thousands, or low single-digits millions. However, if that's wrong, if population gets large enough and regional scripts become a thing... then yeah, we'd run out, but let's hope not -- Unicode is complex enough as it is.

> Remember, you need a fairly large number of users for each glyph, otherwise there's no point.

You are forgetting scripts like Linear A and Linear B which Unicode encodes despite only being of use to a few dozen academics or so at a time, and a few dozen (if that) or so (very) historic documents and maybe hundreds of academic papers over time. Popularity/usage isn't a determiner for Unicode encoding of a script. There are plenty of good reasons to encode low usage scripts and glyphs, if they help people communicate when they are used.

But yes, I agree that it is probably a long while until the Astral Plane runs out, even encoding every small academically interesting thing along the way. I just don't feel we can discount the possibility that the Astral Plane won't be the last plane simply because we can't imagine today what the "Imaginary Plane" would even contain.

I'm not forgetting those or any other historical scripts. I mentioned those. My comment about the number of users of glyphs was in the context of future new glyphs. The supply of old glyphs is pretty limited, so it's easy to accept them all even if the number of users is minute :)

I think you may be confused about the meaning of "plane" in Unicode.

A plane is a contiguous set of 65,536 (216) code points. The lowest-numbered plane was originally the only one; now it's simply Plane 0 (officially, the "Basic Multilingual Plane"). "Astral Plane" is a collective term to refer to planes beyond that, of which there are now 16, for a total of 17 planes.

Unicode currently promises to cap at 17 planes because that's the limit of what UTF-16 can encode with surrogate pairs; anything beyond Plane 16 (officially, "Supplementary Private Use Area-B") would require some other scheme to encode in 16-bit units.

Unicode does not use anywhere near the amount of space even those 17 planes represent; you could encode seven full copies of current Unicode in the unassigned and non-reserved space available, and still have room left over (eight full copies if you un-reserve Planes 15 and 16).

If we ever truly needed more than 17 planes, we'd have trouble trying to do UTF-16, but that's a problem with UTF-16, not with Unicode. UTF-8 as currently defined would run into trouble beyond 32 planes, but we'd either abandon it for something else, or bolt on some inelegant hack to extend UTF-8 beyond four bytes.

So other than maintaining compatibility with existing encodings, there is nothing about Unicode's design that forbids tacking on more planes if and as needed. The hard part, as with so many things in programming, was going from "there's only one" to "there's more than one". If we ever needed enough planes to exhaust both UTF-16 and UTF-8.

One very obvious jest is that at the current rate of emoji expansion it may be inevitable to open up the next plane just for emoji.

To put it in perspective, the total set of emoji in Unicode 11.0, which are spread out across multiple blocks in different planes due to historical reasons, add up to less than 2% of the capacity of a single plane, and 0.11% of the available space in a 17-plane Unicode.

The scheme used in UTF8 can encode up to 42 bits with a single start byte, the last start byte being 0xFF followed by 7 bytes of the form 10xxxxxx.

Hence I said "UTF-8 as currently defined".

You could produce something that uses the same basic scheme as UTF-8 (using the leading byte to indicate the total number of bytes used for the code point), but it would not be UTF-8 as we know it (which caps at four bytes per code point), and different encoders/decoders would need to be developed.

I appreciate the technical description of a plane. I was indeed using the more colloquial "plane [collection]" as shorthand for "collection of planes" as in the "Astral Plane [collection]".

In practice most applications that require a chars(str) function can get away with returning the wrong result for things outside the BMP, as opposed to UTF-8 where you need to start caring as soon as you hit words like "café".

Even if you you do require chars(str) for large strings outside the BMP those characters were so rare before emojis that you could waste a single bit on "contains any non-BMP?" and almost always do the work in O(1) time as opposed to O(n) for UTF-8.

Sorry, that just generates garbage when dealing with things outside the BMP. That can be a lot more common than you think. E.g., when dealing with Chinese characters in a context where unification is not welcomed (e.g., in China).

Yes, you're right that it generates garbage, but that's besides the point.

The point is that a huge number of programmers, especially in the 90s and early 00s would argue for UTF-16 on the basis of it being a fixed width encoding in practice. Maybe they didn't know that it actually wasn't, or they knew and didn't care because they never had to deal with anything outside the BMP.

The overlap between Windows programmers producing software for e.g. in the U.S. or European market and those that would have ever encountered a non-BMP used to be tiny until emojis came along.

So yes, while not in theory, in practice you could get away with treating UTF-16 like fixed width encoding like UCS-2 for a huge number of applications where you could reap the benefits of constant-time chars(str) and charoffset(str, N).

The garbage is super annoying. Please stop. Human scripts are O(N), too bad. You can build indices (must, for large documents), but you can't really avoid this being O(N).

And we're not even talking about normalization.

People get upset about these things and blame Unicode, but the problems are not with Unicode -- they are semantics problems with our scripts that Unicode deals with about as well as can be hoped for.

The only thing I'd remove from Unicode is pre-compositions and the associated normal forms NFC and NFKC. But note that that wouldn't remove the need for normalization.

What are the applications that don’t need to care about decomposed characters when doing counting characters? Twitter’s character limit does, for example. https://developer.twitter.com/en/docs/basics/counting-charac...

> what are their arguments?

It’s often faster. When you process non-English text in UTF8, branch prediction fails all the time because all characters have random size from 1 to 3 bytes. When you process non-English text in UTF16, branch prediction predicts all the time: surrogate pairs are extremely rare so the 99.9% of the stream is 2 bytes/character.

> it's not compatible with ascii and C-strings like UTF-8

In modern C++, wchar_t is a built-in fundamental type: http://www.cplusplus.com/reference/cwchar/wchar_t/

> In modern C++, wchar_t is a built-in fundamental type: http://www.cplusplus.com/reference/cwchar/wchar_t/

But with the slight problem that it actually means UTF-32 on everything on non-Windows (there's char16_t (C++11) and char32_t (C++20), but somewhat obnoxiously, they're new types, so you need explicit casts for pointer conversion)

It's even worse - it means "wide character", which is not necessarily Unicode. E.g. on all BSDs, if your locale is something like zh_TW-Big5, then wchar_t stores Big5 codepoints! You can use it to store UTF-32, of course, but all the wcs* standard functions won't treat it as such.

Now, there's __STDC_ISO_10646__, which, if it is defined by an implementation, means that wchar_t is guaranteed to be a Unicode codepoint. So e.g. glibc defines that, and on Linux you can therefore assume wchar_t as UTF-32 regardless of locale. But BSDs don't. And on Windows, nothing that uses 16-bit wchar_t can use that define, since 16 bits aren't wide enough to be conforming.

Wow, branch prediction is actually a pretty good point I hadn't thought about.

It's common, though, to decode to an internal 16- or even 32-bit representation. Combined with clever UTF-8 decoding state machines the overhead is minimal, whether you do this all at once or in a streaming, segmented fashion. Rare is the circumstance where you can just load UTF-16 directly into an array and go to town without preprocessing. In most application scenarios you need to translate to a normalization form (i.e. NFC, NFD, etc), which you can simply combine with your UTF-8 decoding. Such tight loops of arithmetic code are blazingly fast, and performance can improve dramatically by batching these steps separately rather than intermixing different phases of text processing in your application.

Perl6 decodes to NFG form which provides a 1:1 mapping between characters and scalar values.[1] That makes Unicode string munging almost as easy as ASCII.[2] But it requires always preprocessing and intern'ing your strings.

[1] Multi-codepoint graphemes are dynamically assigned a unique internal codepoint.

[2] I say almost because some Unicode text rules, like those for word and paragraph breaks, can never be implemented as simple as in ASCII.

Correct me if I'm wrong:

UTF-16 can be faster if you're iterating through characters, but for parsing iterating through bytes is usually sufficient. It's fine to use something like plain old strchr to search for a { or a < in a UTF-8 string. Characters only get really important when you want to print the string.

> It's fine to use something like plain old strchr to search for a { or a < in a UTF-8 string.

Correct.

> only get really important when you want to print the string.

Print, split into fixed-length pieces, render, layout, typeset — GUI apps do that a lot with their strings.

Even web browser do, probably that’s why JavaScript strings are UTF-16 despite web in general is mostly UTF-8, see section 6.1.4 “The String Type” on page 67: http://www.ecma-international.org/publications/files/ECMA-ST...

Very true, but once you're at the level of complexity where you're rendering a GUI are you really concerned about cache invalidation due to incorrect branch prediction iterating strings?

GUI rendering is very expensive for lots of reasons, but I don't believe that's one of them - even in the UTF-8 case. You're ultimately either drawing thousands of pixels or communicating over a (relatively compared to the CPU cache) slow bus with the GPU.

> You're ultimately either drawing thousands of pixels or communicating over a (relatively compared to the CPU cache) slow bus with the GPU.

In modern software, GPU draws pixels. Before it does that, CPU lays out these glyphs. Because GPUs are ridiculously fast these days, the layout step is typically slower than painting.

I use MS edge browser. The built-in profiler said this comments page took 14ms to layout and only 6ms to paint. This page contains just a few tiny images, the majority of the content is text. I think the layout step spent most of the time iterating over the characters, looking up glyphs and measuring various blocks of text on this page.

As pointed out elsewhere, you are wrong because UTF-16 'characters' can still be comprised of compounded elements. (Base character plus additional composition elements to create a final character.)

In simple terms, you can't just treat it as an array to index to any given character.

There's also all the detriments that still apply. (Including UTF-16LE / UTF-16BE, BOM, not being able to concatenate two valid string sequences (blind) and always have a valid result, etc.)

You're also incorrect, or at least not correctly describing the search operation.

The assumption about finding specific characters (for example, command flags) in a string or array of strings is also... complicated.

As an example: https://en.wikipedia.org/wiki/Halfwidth_and_fullwidth_forms

For /command/ flags the half and full width equivalent characters might want to be 'folded' back over the traditional ASCII namespace.

Sometimes a specific loss of precision can be desirable.

> As pointed out elsewhere, you are wrong because UTF-16 'characters' can still be comprised of compounded elements. (Base character plus additional composition elements to create a final character.) In simple terms, you can't just treat it as an array to index to any given character.

I am aware of that having implemented UTF-8 myself, and I thought I was fairly clear in my comment about characters vs. bytes.

> The assumption about finding specific characters (for example, command flags) in a string or array of strings is also... complicated. As an example: https://en.wikipedia.org/wiki/Halfwidth_and_fullwidth_forms

Also aware of that. But it only matters if you're actually searching for those characters. Parsing something like XML or JSON, therefore, is as easy with UTF-8 as it would be with ASCII, because you're only looking for characters like < or { which are the same byte values in UTF-8. You don't need to worry about accidentally finding a continuation byte because UTF-8 sets high bits on continuation bytes for exactly this reason.

Is this UTF-16(BE|LE) and UTF-8 mixed?

I'm unclear about the context you're proposing. If you're trying to match for a Unicode character in either UTF-16 bytestream dialect then determining the matching format first is important. The mixed format just seems too crazy to touch.

I think it's also more space efficient when dealing with foreign languages primarily. I agree that UTF-8 is the better default, but that's one of the most compelling arguments in favor of it (aside from it being the default on Windows).

Even this argument isn't great, though, because a lot of the time it doesn't matter. Like, for HTML, the majority of the time the UTF-8 version will be shorter regardless because of all of the single-byte characters used in the markup.

I think it's also more space efficient when dealing with foreign languages primarily.

For plain text, yes. UTF-16 covers the Basic Multilingual Plane in 2 bytes per character, UTF-8 covers the BMP in at most 3 bytes per character. However, since most (web) content is dominated by markup in ASCII, which is 1 byte per character in UTF-8 and 2 bytes per character in UTF-16, UTF-8 typically wins even with Asian scripts.

Taking some random zh Wikipedia page in UTF-8:

    curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -c
    78543
    curl "https://zh.wikipedia.org/wiki/威蘭運河" | wc -m
    71003

So, that's 1.11 bytes per character on average, compared to at least 2 bytes per character if it was encoded in UTF-16.

(Of course, adding compression is going to diminish the differences anyway. A very primitive compression algorithm in such cases would be recoding your UTF-16 to UTF-8 :p.)

  curl -Ss "https://zh.wikipedia.org/wiki/威蘭運河" | iconv -t utf-16 | wc -c
  142008

There are only 3799 non-ASCII characters on that page:

  curl -Ss "https://zh.wikipedia.org/wiki/威蘭運河" | tr -d '\1-\176' | wc -m
  3799

Compared with 67204 ASCII ones (tr -cd).

The 142008 is to be expected. The page has 71003 code points, 2 bytes per code point (I guess that zh.wikipedia.org is simplified Chinese and normally only uses characters from the BMP) gives 142006 bytes.

I'd guess that the two remaining bytes are the byte order mark.

> I agree that UTF-8 is the better default

Better default for what? I agree UTF-8 is better default for HTML sent over network. For different context, the better default can be different.

> side from it being the default on Windows

Also default in Java (including Android userland), C#, iOS & OSX, python 3 (unless compiled with UTF32 support but the default is UTF16), Symbian, OpenOffice, QT, most XML parsers (e.g. Xerces), and many other languages, libraries and frameworks. Even JavaScript uses UTF16 strings.

Better default for everything. If you have particular needs, then you change from the default. This is why 'default values' exist.

> Better default for everything.

Software industry is huge and the requirements are dramatically different in different areas. What’s good default for a bash script is not necessarily a good default for OSX desktop app, or Linux web server.

When you’re working on native rich GUI, UTF16 is often better because majority of higher-level languages, libraries and frameworks use it, even on Linux (QT, Android). If you’ll pick something that’s not native, you’ll waste substantial time, both development and CPU, converting these strings back and forth for absolutely no value.

Sometimes you have to because you’re integrating different things together, e.g. UTF16 Android’s Java with UTF8 Android’s C++: http://banachowski.com/deprogramming/2012/02/working-around-... But quite often you are good just picking whatever’s the simplest option and not doing any of that.

> python 3 (unless compiled with UTF32 support but the default is UTF16)

Not true since Python 3.3. It now uses latin1/ucs-2/ucs-4 based on string contents (smallest encoding that can represent all codepoints).

> python 3 (unless compiled with UTF32 support but the default is UTF16)

The default is UTF-32 on Linux, and UTF-16 on Windows and macOS - simply so as to match the platform convention.

> python 3 (unless compiled with UTF32 support but the default is UTF16)

Not the case since PEP 393 / Python 3.3.

Also pretty much all Linux distros used wide builds.

> Also default in Java (including Android userland), C#, iOS & OSX, python 3 (unless compiled with UTF32 support but the default is UTF16), Symbian, OpenOffice, QT, most XML parsers (e.g. Xerces), and many other languages, libraries and frameworks. Even JavaScript uses UTF16 strings.

Most of these don't actually use UTF16 but UCS2 with surrogates (sometimes nicknamed WTF16).

> Most of these don't actually use UTF16 but UCS2 with surrogates

Most of these actually use UTF16.

.NET has System.Globalization.StringInfo class, Java has string methods like String.codePointCount. Operator [] returns 2-byte code points in these languages because backward compatibility. Newer languages don’t need to be backward compatible see e.g. https://docs.swift.org/swift-book/LanguageGuide/StringsAndCh... but their internal format is still UTF-16. If you need to interop with any of these languages, your life will be much easier if your C++ code uses UTF-16 as well.

> Most of these actually use UTF16.

All of .Net, Java and Javascript[0] allow unpaired surrogates, meaning none of them uses UTF-16. I expect the same happen for Symbian, OpenOffice, Qt, … So did Python on pre-FSR narrow builds.

Swift is migrating to UTF-8 as part of its ABI stabilisation: https://forums.swift.org/t/string-s-abi-and-utf-8/17676

[0] unclear for ObjC/NSString but given the string interface leaks it being an array of utf-16 code units I'd expect the same

edit: https://stackoverflow.com/a/33558934/8182118 indicates that both NSString and Swift strings can contain unpaired surrogates, so not UTF-16 either.

> meaning none of them uses UTF-16

Depends on your definition of “uses”, and I disagree with yours.

It’s always technically possible to make invalid data. For example, you can write a web app that will send Content-type: application/json and invalid utf-8 in the response.

Yes, technically these programming languages allow unpaired surrogates in strings. This is not necessarily a bad thing, there’re valid uses for such strings. For example, you can concatenate strings without decoding UTF16 into code points, i.e. the code will be slightly faster, but while you’re concatenating, in some moment of time the destination will contain an unpaired surrogate.

I hope you agree golang uses utf-8 strings. Despite that, the unicode/utf8 standard package has ValidString function, which means you can still create invalid strings in go: https://golang.org/src/unicode/utf8/utf8.go Does it mean golang uses wtf-8 instead of utf-8?

> Depends on your definition of “uses”, and I disagree with yours.

It really does not, unless you're also disagreeing with the definition of "is". Unpaired surrogates are not valid in a UTF-16 stream. If there are unpaired surrogates it's not UTF-16. If the system normally allows and generates unpaired surrogates, it's not dealing in UTF-16.

> It’s always technically possible to make invalid data. For example, you can write a web app that will send Content-type: application/json and invalid utf-8 in the response.

And that will blow up in the client, because your shit system has sent it garbage.

> Yes, technically these programming languages allow unpaired surrogates in strings.

Making them not-UTF-16.

> This is not necessarily a bad thing, there’re valid uses for such strings.

Like taking a pile of garbage and making it into a bigger pile of garbage.

Which is not relevant to to the issue at hand: such piles of crap are not UTF-16.

> For example, you can concatenate strings without decoding UTF16 into code points

That's not actually an example of your claims, you can concatenate valid UTF-16 without decoding it into codepoints as well.

> I hope you agree golang uses utf-8 strings.

I most certainly do not, why would you hope I agree with an obviously incorrect statement, what is wrong with you?

> Despite that, the unicode/utf8 standard package has ValidString function, which means you can still create invalid strings

Well you've got cause and effect reversed but yes you can create non-utf8 go strings, making Go's "strings" not utf8 at all.

> Does it mean golang uses wtf-8 instead of utf-8?

No, WTF-8 is something well-defined[0].

Golang's "strings" are just random bags of bytes, much like C's, and they're no more utf-8 than C's: you may assume they are of whatever encoding you're interested in but with no guarantee whatsoever that's actually the case and your assumptions may well blow up in your face.

[0] https://simonsapin.github.io/wtf-8/

Yes, unpaired surrogates are invalid UTF16.

No, just because languages allow representing invalid strings in their type system doesn’t mean they use some other encoding.

Ability to represent invalid data is often a good thing. English language allows representing all kind of garbage, but this is OK, see e.g. Jabberwocky by Lewis Carroll.

There are programming languages who enforce strings encoding and other constraints with strict type systems and/or runtimes. They are only practical for very limited set of problems. The majority of real-world software has to deal with invalid data: compilers do because users type invalid programs, any sufficiently complex systems do because they receive data from external components written in other languages or from I/O. I think not being able to represent invalid strings is a bug, not feature.

Mainstream i.e. practical languages allow representing invalid strings, provide functionality to validate & normalize them, and often raise runtime errors when you try to use them in a way that makes this a problem. For example, .NET throws an ArgumentException saying "Invalid Unicode code point found at index ##" when you try to normalize an invalid string.

> AFAIK the only advantage of UTF-16 is that it's a little more efficient when encoding text written in certain scripts.

and IIRC east Asian languages (which benefit the most) almost only use UTF-8

And almost all actual uses are cases where markup (be it HTML, OOXML, ODF) are sufficiently common that the total file size ends up being larger using UTF-16 than UTF-8.

I don't know if this is still the case, but as of a few years ago, there was clear evidence that the Japanese would rather have their text corrupted via mojibake than silently use UTF-8. Of course, that's about avoiding Unicode entirely rather than preferring UTF-16 over UTF-8.

TIL about "mojibake"

(I visited Japan for 2 weeks for the first time about a month ago; had a great time)

Do you know why that is? Why don't they like Unicode?

A couple reasons I'm aware of:

- Han unification. Chinese and Japanese characters that have the same etymology but are written differently are sometimes assigned to the same codepoint, and it's up to the font to distinguish. And application designers... tend not to detect the language and change the font accordingly. This leads to Japanese text being rendered in a Chinese font.

- Some Han de-unification has happened in Unicode now. Great. Which version should fonts and encodings support?

- Nobody actually knows the complete de facto mapping between Shift-JIS and Unicode. Yes, there are standards and ICU and Python modules and stuff; they're incomplete. This leads to data loss surrounding rare characters.

(Tell me you've got such a mapping and I'll give you some strings I found in the wild to decode with it.)

>Han unification.

I am still hoping some day down the road we can fix this without overstepping on each other's culture and fonts / glyph.

One legitimate concern I know of for email is that you could still get Japanese mobile phones as late as 2008 that didn't support UTF-8. People setting their mail filters to autotrash UTF-8 has also definitely happened (pro tip: never attempt to filter by charset; if you really want to, filter by script instead).

I believe another major issue is that a fair amount of software sets language environment for things such as fonts based on charset. Charsets such as ISO-2022-JP, GBK, and Big5 give you a clear clue as to what the underlying language is, but UTF-8 just tells you "here is CJK ideographs, good luck!" On the other hand, I've never really heard of any opposition to Unicode from Chinese or Korean locales, so I suspect a large part of it is just stubborn tech entrenchment in Japan as opposed to strong technical reasons.

> east Asian languages (which benefit the most) almost only use UTF-8

China, a major user of East Asian languages, uses GB2312, certainly not UTF-8.

UTF-8 assigns 3 bytes to Chinese characters so it can use the 2-byte space for European characters; that's not exactly a big selling point when you're using East Asian languages.

Reply on news.ycombinator.com