https://tools.ietf.org/html/rfc2482
But it was deprecated later on:
(Somewhat ironically, the previous thread on that mailing list is about the struggles of case folding in a general fashion across multiple language scripts[2])
Edit: I also found [3], which offers the following:
----
- Most of the data sources used to assemble the documents on the Web will not contain these characters; producers, in the process of assembling or serializing the data, will need to introspect and insert the characters as needed—changing the data from the original source. Consumers must then deserialize and introspect the information using an identical agreement. The consumer has no way of knowing if the characters found in the data were inserted by the producer (and should be removed) or if the characters were part of the source data. Overzealous producers might introduce additional and unnecessary characters, for example adding an additional layer of bidi control codes to a string that would not otherwise require it. Equally, an overzealous consumer might remove characters that are needed by or intended for downstream processes.
- Another challenge is that many applications that use these data formats have limitations on content, such as length limits or character set restrictions. Inserting additional characters into the data may violate these externally applied requirements, and interfere with processing. In the worst case, portions (or all of) the data value itself might be rejected, corrupted, or lost as a result.
- Inserting additional characters changes the identity of the string. This may have important consequences in certain contexts.
- Inserting and removing characters from the string is not a common operation for most data serialization libraries. Any processing that adds language or direction controls would need to introspect the string to see if these are already present or might need to do other processing to insert or modify the contents of the string as part of serializing the data.
----
Other than #3 (the one about string identity), I find these wholly unpersuasive. And even #3 isn't even that great a reason considering that programmatic processors have to deal with that issue anyway due to case folding.
[1] https://www.unicode.org/mail-arch/unicode-ml/y2010-m11/0039....
[2] https://www.unicode.org/mail-arch/unicode-ml/y2010-m11/0038....
The problem was never gazillions of code pages, but our inability to write C to deal with that amount of complexity circa 1990.
With modern machines, and good programming languages with good type systems, I absolutely think we could store a language per string, and concatenate into a polylinguistic rope if needed.
This would hopefully push us away from stringly-typed crap in general.
I wish you would give an example instead of just proclaiming crapness. You know, so we n00bs can learn something.
;)
Sorting of strings is language dependent, but I don't know that there's a defined order for mixed language lists, so I guess user's context works if you're sorting for user purposes, but if you're sorting for machine purposes, you better not use the locale aware sort without telling it a hardcoded locale that doesn't change between localization library versions.
The flaw in Unicode is that it punts on the intrinsic complexity---pretending that codepoints have language-independent, plain-text, semantic meaning.
A couple of threads that have molded my views over time:
I can't write my name in Unicode https://news.ycombinator.com/item?id=9219162 (Specifically these two comments https://news.ycombinator.com/item?id=9220530 and https://news.ycombinator.com/item?id=9220970)
Why isn't the external link symbol in Unicode? https://news.ycombinator.com/item?id=23016832
> Pretending "plain text" isn't an oxymoron
FTFY :)
E.g. lookalike charaters, and social engineering by using a vs а. (One is Cyrillic). I don't want to even define "a == а". I want Latin and Cyrillic to be different types of characters, and that expression to be ill-typed.
This solves the Turkish problem, where the upper case I is two different charters in two different types (Turkish Roman script?), and the case folding functions likewise have disjoint types.
How do you concatenate English and Ру́сская text, and what is the type of this sentence?
If you want to build a polylinguistic rope you can certainly do that with Unicode, but you won’t have solved anything because language alone without context doesn’t really define many of the operations you’re describing.
The answer is usually the same as “doctor it hurts when I...” — stop doing it. Stop manipulating user input without context. Stop trying to limit user visible strings by character count, use pixel width in the rendered font. And so on.
Sure it does; the Unicode Character Database includes fields for the lowercase, uppercase and titlecase mappings. But it also acknowledges that these are just default mappings, and may need to be tailored for specific languages/locales.
If all that good effort went into something along the lines I am describing, where languages, or at least scripts, cannot be arbitrary mixed at the character level, I think we would have an even better result with the same level of effort.
That said, any "multicode" better describe the interopt with "unicode" in great detail for practical reasons. Still this is the "FFI", and one can be careful to let it not muddle things by e.g. not allowing every unicode string to be imported without additional metadata.
I am happy to define how to translate Multicode to Unicode, but I wouldn't want any of the internal notions of Multicode to be defined in terms of that translation.