The saddest thing about Python 3 is that they made a breaking change to do Unicode "right" and still did it wrong. The right way to do Unicode is the way Rust does it: UTF-8 in memory and no (safe) API to introduce UTF-8 invalidity.
UTF-32 is wrong, bwcause it's wasteful and still doesn't accomplish what people naively expect due to grapheme clusters potentially taking more than one UTF-32 code unit.
> UTF-32 is wrong, [because] it's wasteful and still doesn't accomplish what people naively expect due to grapheme clusters potentially taking more than one UTF-32 code unit.
Out of curiosity, is there a correct way to encode unicode that doesn't involve this level of surprise? I thought that this was still an unsolved problem at this point.
Then you offer a data structure that lets you perform O(1) or O(logn) operations on sequences of single-character strings.
If it's read-only you could make it just be an index, blah blah the details don't matter a lot, the point is you can make something that's both correct to grapheme clusters and probably more space-efficient than UTF-32 despite the extra data.
And then the encoding inside the character strings isn't particularly important, but might as well use UTF-8.
-
Either that or make yourself a hilariously inefficient format based on:
UAX15-D3. Stream-Safe Text Format: A Unicode string is said to be in Stream-Safe Text Format if it would not contain any sequences of non-starters longer than 30 characters in length when normalized to NFKD.
Who's with me on 128-byte characters.
On the human side, people do not like to stare at glaring mistakes all the time, if they can do something about it -- having weighed the cost associated with the change.
In 20 years if python is still a thing, it likely will not be because it was designed right from the get go. It will be because it changed enough to stay relevant despite the fact that some changes may have left incompatibilities in their wake. Users have to live with that.