Hacker News Reader

Comment by LegionMammal978 | Hacker News Reader

The problem is that UTF-8 has the ability to detect and reject partial characters at the start of the string; this encoding would silently produce an incorrect character. Also, UTF-8 is easily extensible already: the bit patterns 111110xx, 1111110x, and 11111110 are only disallowed for compatibility with UTF-16's limits.

View on HN

How often are stream truncated at the start? In my career, I've seen plenty of end truncation, but start truncation never happens. Or, to be more precise, it only happens if previous decoding is already borked. If a previous decoding read too much data, then even UTF-8 is borked. You could be decoding UTF-8 from the bits of any follow-up data.

Even for pure text data, if a previous field was over-read (the only plausible way to have start-truncation), then you probably are decoding incorrect data from then on.

IOW, this upside is both ludicrously improbable and much more damning to the decoding than simply be able to skip a character.

It happens all the time when you're working with substrings. Under your encoding, "é" would be a substring of "ჩ":

           10000011 11101001 U+E9 "é"
  10000001 10000011 11101001 U+10E9 "ჩ"

This would make it incompatible with many existing processes that already handle text, which was one of the goals of UTF-8.

it's nice seeing my native language's character: ჩ

Imagine you plug a dumb terminal into an RS232 port. The computer may have already sent some data. Or it may be in the middle of sending log messages to the console as you plug in.

With a code like this, your dumb terminal can be built so it just automatically works. Without it, the terminal may be off by some number of bytes, and you have to have some sort of other (probably manual) synchronization procedure.

Similarly, imagine you've dialed in over modem to a remote system. You get some errors or lost text due to noise on the phone line. Or some bytes are lost due to bad RS232 flow control or no flow control. Your terminal emulator would be able to recover.

There might also be uses where text is broadcast, such as in TV closed captions, although maybe those have their own framing so you know where strings begin.

Reply on news.ycombinator.com