Hacker News Reader

Comment by Animats | Hacker News Reader

That was set up when Microsoft and Java had standardized on UTF-16. So this can represent the UTF-16 subset of Unicode, which is Unicode Plane 0, the Basic Multilingual Plane (BMP). The higher-numbered "astral planes" of UTF-8 were rarely used. All modern languages with a significant user base are covered in Plane 0. Plane 1, the Supplementary Multilingual Plane, with Cretan Linear B, Egyptian hieroglyphics, and such, was seldom needed. Few people had fonts for those, anyway.

Because of the way UTF-8 is encoded, it takes 3 bytes to represent the UTF-16 set. That's because it only takes one byte for ASCII characters. Hence, 3-byte MySQL data.

Emoji, though, were put in Unicode Plane 1. That's where mass demand for the astral planes came from. More four byte UTF-8 characters started showing up in data.

View on HN

A UTF-16 "subset" of Unicode doesn't exist. all UTF-* encodings are just that: encodings. They can all represent the entire Unicode character set. (Except for UTF-7, which never was an official standard).

What you're referring to is UCS-2 [0], which is UTF-16 without support for the high surrogates that make up the rest of the Unicode character set. But to imply that UTF-16 is a subset of Unicode is just not true.

[0] https://en.wikipedia.org/wiki/UCS-2

That they can all represent all of unicode is true, but they're not "just" encodings either. The current code space of unicode is built around UTF-16. That's why there are 17 planes.

And why surrogates are a thing, and why code points and scalar values are different concepts.

I hate UTF-16. It’s the worst thing that ever happened to Unicode. It ruined Unicode.

Code points, scalar values, glype, surrogates, encodings

Unicode should be called multicode really. If we want something "uni-" then all of them should better be unified to one encoding. Hopefully UTF-8 do the job.

There's so much wrong with this answer. It completely confuses Unicode, with Unicode encodings, and with non-Unicode encodings.

> So this can represent the UTF-16 subset of Unicode

There is no UTF-16 subset of Unicode. UTF-16 has surrogate pairs, it can represent all of Unicode. You are talking about UCS-2, a format which has no surrogate pairs.

> That was set up when Microsoft and Java had standardized on UTF-16

Java standardized on UCS-2 initially (which is what you're talking about). In the mid 2000s they moved to actual UTF-16.

> The higher-numbered "astral planes" of UTF-8

"astral planes" are not a UTF-8 concept. They're a Unicode concept. A Unicode code point has 6 hex digits. The plane is just the first 2 of those 6 digits. So code points 0000– FFFF are in the BMP (Basic Multilingual Plane), etc. People nickname planes above the 00 plane, BMP, astral planes.

> Plane 1 ... was seldom needed. Few people had fonts for those, anyway.

So "rarely", that the majority of humanity writes in languages that aren't in BMP. CJK (Chinese, Japanese, Korean and sometimes Vietnamese) languages need characters outside of BMP.

> Emoji, though, were put in Unicode Plane 1. That's where mass demand for the astral planes came from. More four byte UTF-8 characters started showing up in data.

That's totally untrue. The demand always existed by the majority of humanity, you just never cared about this massive problem until it impacted you.

> That's totally untrue. The demand always existed by the majority of humanity, you just never cared about this massive problem until it impacted you.

I think you swung so far in the other direction you also landed in untrue territory. Originally the intent was to encode all characters in "modern use" and 16 bits was probably enough to do that for the entire world.

Also CJK is about one quarter of the world population.

"originally" is 20 years ago. People realized this mistake almost immediately before fixing it in Unicode 2.0. The mistake existed for something like 5 years last millennium.

There's no excuse for not taking seriously a large part of the world's communication needs in 2022 as something esoteric that no one but people in 5000BC needed.

UTF-16 can represent the full range of Unicode codepoints by using couples of surrogates

Sort of. Applications using UTF-16 have to be aware of pairs at the application level. Many are not.

This isn't a consequence of using UTF-16 as such - Java, .NET etc could totally have an API around UTF-16 strings that handles surrogate pairs. The problem, rather, is that those languages introduced a 16-bit type that they called "character", even though it wasn't even a Unicode codepoint. And then used that type throughout all string APIs, including strings themselves (indexing etc).

In .NET land you're now supposed to use https://docs.microsoft.com/en-us/dotnet/api/system.text.rune instead. It transparently handles surrogate pairs, so the app needn't be aware of anything - and yet the internal encoding is still UTF-16.

You're right. Replace UTF-16 with UCS-2 and the comment sounds at least slightly more correct.

Technical amendment: UTF-16 can represent the full range of Unicode scalar values with surrogate pairs. Code points includes the surrogates U+D800–U+DFFF, scalar values don’t. Like all other Unicode encodings, UTF-16 cannot represent surrogates.

That’s where the real problem lies: almost nothing that uses UTF-16 actually uses UTF-16, but rather potentially ill-formed UTF-16.

Reply on news.ycombinator.com