How to Create a UTF-16 Surrogate Pair by Hand, with Python

The biggest takeaway/shock for me in this article is the fact that JSON string literals can't contain escaped characters outside the Basic Multilingual Plane (i.e. whose code points are greater than U+FFFF).

Quoting RFC 8259:

   To escape an extended character that is not in the Basic Multilingual
   Plane, the character is represented as a 12-character sequence,
   encoding the UTF-16 surrogate pair.

So in order to encode U+1F914 THINKING FACE you can either do

  {"text": "<thinking face>"}

(Emoji omitted bc of HN) or

  {"text": "\ud83e\udd14"}

but not

  {"text": "\u1f914"}.

This seems to be a relic from ECMAScript which (iirc, only skimmed it) assumes UCS-2/UTF-16. From that perspective it makes a lot of sense but it steel feels a little icky to me having surrogates referenced in a standard that is supposed to be UTF-8 only :)

Surrogate pairs are such a useless thing to exist.

A unicode character is just a number (of up to around 21 bits). The encoding of this number, and the unicode value itself, can (and should) be made completely independent. Note how UTF-8 did not introduce any useless values in unicode itself. That's because this code is sane (if a bit inefficient due to over-focus on error correction) and correctly makes its encoded byte values independent of the numeric value being encoded.

For UTF-16, initially they thought 16 bits would be enough, and then they had to hack something on top of it to make it variable length. Fair enough, but even that hack, they could have done without introducing useless unicode values: they could have defined the variable length 16-bit code to use the unused codes to indicate variable length, and then add/subtract values while decoding it, that is only use the unused UTF-16 values to tell how to use the remaining UTF-16 bits to take existing consecutive values from unicode. This would have been not a bigger mess than surrogates are, be transparent to users (since the decoder deals with the math), and left the mess inside of UTF-16 only (which at this point they should have known was already game over since the planned 16 bits didn't fit) instead of infecting the rest with its mess.

So not only is UTF-16 the ugliest encoding ever by being the worst of both worlds (variable length and yet bigger than a byte and not ascii compatible) and damages both windows (ruined filename loading in multiplatform C libraries, horrible "wstrings" in windows code) and java/javascript (UTF-16 in the language while the internet they interface with are UTF-8, exposing programmers to "surrogates"), but in addition in the making of this ugly code they managed to also uglify unicode itself, and UTF-8 (which has to deal with the fact it can refer to surrogates), and any other future encoding of unicode too!

> As another data point, the node.js interpreter chooses to print the � replacement char instead of raising an exception.

You can do the same in Python:

    >>> print(s.encode("utf-8","replace").decode("utf-8"))
    ?

Or do this:

    import sys
    sys.stdout.reconfigure(errors="replace")

Alternately, start python with:

    PYTHONIOENCODING=utf8:replace

You can also use "ignore" to drop the illegal character entirely.