Comment by neild | Hacker News Reader

Comment by neild | Hacker News Reader

Parent Full thread

It's exceedingly rare that you need a sequence of codepoints. As evidence, consider that the native string types in many languages with pervasive Unicode support, such as Java and C#, do not provide a codepoint sequence.

Programs generally need to deal with textual data at one of three levels:

1. Manipulate strings and substrings. e.g., "does string x contain substring y"? Byte sequences of utf8 data are fine for this. You may need to normalize the data first, but that's true of codepoint sequences as well.

2. Deal with a small number of specific ASCII codepoints--e.g., the filename separator character. Again, utf8 byte strings are fine for this.

3. Deal with glyphs, including glyphs composed of multiple combining characters. You need to iterate to assemble the glyphs, so a codepoint sequence offers no advantages over a byte sequence of utf8 with a codepoint iterator.

A sequence of bytes is just fine as a native string type. Store Unicode data as utf8-encoded byte strings. Provide easy iteration over codepoints in utf8 strings, transcoding for I/O, normalization functions, etc. Call it a day.

The problem isn't access to individual codepoints, it's mixing strings with different encodings without any way of tracking the encoding or knowing that you're doing something wrong.

UTF-8 everywhere works great when you can enforce it. On the level of an individual project, you can enforce it. On the level of a language ecosystem, you can't, and you need to. Otherwises you end up with some libraries who assume their internals are UTF-8 encoded strings, some libraries that assume they are ASCII strings, some libraries that assume the caller is handling encoding issues and will take care of ensuring that it's all UTF-8, some that make no assumptions at all and carry around the encoding everywhere, and some that just haven't thought about the problem and silently break when you use them with data that came from other libraries.

It's interesting that Go and Java - both languages with mature Unicode handling - still have a distinction between uninterpreted bytes and UTF-8 or UCS-2 text. In Go, you have separate []bytes and string types, even though Go strings are just UTF-8 byte sequences. In Java, you have byte[] and String. The problem is not a technical one of how to represent strings, it's a social one of how to get all the library authors on a language to agree on a convention of how to handle encoding.

> it's a social one of how to get all the library authors on a language to agree on a convention of how to handle encoding.

Ok, but the solution isn't to make everyone rewrite their libraries by holding the future ransom if they do not.

It could have been real simple: let python track encoding when it's known and throw an exception if they mix oldstr(unknown) with newstr(py3).

In py3 modules newstr is str, and in py2 modules oldstr is str. Now you only have to fix the bits where they mix and the programmer can always choose to fix it in the new code.

This is the first time I've ever seen someone write with an understanding of combining characters, glyphs, codepoints vs encoding of said codepoints - and yet arrive at this conclusion.

What's the largest codebase you tried a unicode-ification project on? It's a nightmare unless you keep de/encoding as close to the i/o operations as possible.

I can't understand how you've ever found it just as easy to do "string x contain substring y" on bytes vs uc strings. Any case-insensitive test will fail miserably unless you only ever see ASCII input. Then there's sorting and tokenization. Oh god, the sorting bugs...

Even measuring the length of string is a miserable fail. And blind substitution of utf8 bytes horribly mangles the output causing mysterious segfaults or silent corruption.

On a large codebase, programmers can't keep track of what encoding is being used in which parts of the code. Eg. Let's allow the users to specify input file encoding! But our OS does filenames in UTF16-LE. And the Web API is UTF-8... nasty stuff. It's far saner to use character strings everywhere except immediately after/before I/O operations.

Reply on news.ycombinator.com