Hacker News Reader

Comment by Ericson2314 | Hacker News Reader

What this gets right down to is that Unicode is a flawed idea: the meaning/behavior/whatever of characters is insanely dependent on their context.

The problem was never gazillions of code pages, but our inability to write C to deal with that amount of complexity circa 1990.

With modern machines, and good programming languages with good type systems, I absolutely think we could store a language per string, and concatenate into a polylinguistic rope if needed.

This would hopefully push us away from stringly-typed crap in general.

View on HN

Unicode goes to great pains to avoid ascribing any meaning/behavior/whatever to character sets. Because to your point you can’t. Unicode is actually incredibly well thought out. That’s why we have values, code points and grapheme clusters. I don’t think the Unicode standard even defines casing except in the human-readable names ascribed to code points.

If you want to build a polylinguistic rope you can certainly do that with Unicode, but you won’t have solved anything because language alone without context doesn’t really define many of the operations you’re describing.

The answer is usually the same as “doctor it hurts when I...” — stop doing it. Stop manipulating user input without context. Stop trying to limit user visible strings by character count, use pixel width in the rendered font. And so on.

> I don’t think the Unicode standard even defines casing except in the human-readable names ascribed to code points

Sure it does; the Unicode Character Database includes fields for the lowercase, uppercase and titlecase mappings. But it also acknowledges that these are just default mappings, and may need to be tailored for specific languages/locales.

Unicode is well thought out! And that's what makes it hard to critique :). I think it's one of the best-maintained, well-thought out standards there is, but I still think the premise is wrong.

If all that good effort went into something along the lines I am describing, where languages, or at least scripts, cannot be arbitrary mixed at the character level, I think we would have an even better result with the same level of effort.

If you treat Unicode as your backing representation -- a pile of glyphs -- you can build what you're asking on top right?

That is like saying I can take an untyped language and then add types. Sure you can! That said, it's much nicer (to me) to first define the typing rules (static semantics) and then define evaluation (dynamic semantics) only on well-typed programs. This avoids the need to include lots of annoying stuff in the domain. See my other comment, https://news.ycombinator.com/item?id=24180620, for an example of something I rather leave ill-typed.

That said, any "multicode" better describe the interopt with "unicode" in great detail for practical reasons. Still this is the "FFI", and one can be careful to let it not muddle things by e.g. not allowing every unicode string to be imported without additional metadata.

I'm suggesting it's more like layering a programming language on top of assembly. The lower level is the universe of what you can do (in this case, the set of all glyphs) and the higher level is an imposition of specific constraints (in your case, which ones go together).

Languages need not by defined by how they compile. (If they do, we tend to call it "desugaring".) At the very least, they usually compile to multiple ISAs, and none is more definitive than the other.

I am happy to define how to translate Multicode to Unicode, but I wouldn't want any of the internal notions of Multicode to be defined in terms of that translation.

> the meaning/behavior/whatever of characters is insanely dependent on their context

I wish you would give an example instead of just proclaiming crapness. You know, so we n00bs can learn something.

How about: case folding for the letter 'I' is dependent on whether the locale is Turkish or not.

;)

Different languages have different rules for change case (as seen here) or what to do when translitterating to 7-bit ascii, in French, you can mostly drop accents if you need to, in German, you need to transform an umlaut to an e following the vowel. Of course, many languages don't have a way to translitterate to 7-bit ascii.

Sorting of strings is language dependent, but I don't know that there's a defined order for mixed language lists, so I guess user's context works if you're sorting for user purposes, but if you're sorting for machine purposes, you better not use the locale aware sort without telling it a hardcoded locale that doesn't change between localization library versions.

@toast0, @lazulicurio, both of your points seem to illustrate the complexities of the languages, not "...that Unicode is a flawed idea" as the original poster said. AFAIKS this is intrinsic complexity showing itself and does not make any indication of how it should be done correctly, or better.

> both of your points seem to illustrate the complexities of the languages, not "...that Unicode is a flawed idea"

The flaw in Unicode is that it punts on the intrinsic complexity---pretending that codepoints have language-independent, plain-text, semantic meaning.

A couple of threads that have molded my views over time:

I can't write my name in Unicode https://news.ycombinator.com/item?id=9219162 (Specifically these two comments https://news.ycombinator.com/item?id=9220530 and https://news.ycombinator.com/item?id=9220970)

Why isn't the external link symbol in Unicode? https://news.ycombinator.com/item?id=23016832

> The flaw in Unicode is that it punts on the intrinsic complexity---pretending that codepoints have language-independent, plain-text, semantic meaning.

> Pretending "plain text" isn't an oxymoron

FTFY :)

The benefit of looking at languages/scripts in isolation is that the combinatorial explosion of all languages/scripts at once is dodged.

E.g. lookalike charaters, and social engineering by using a vs а. (One is Cyrillic). I don't want to even define "a == а". I want Latin and Cyrillic to be different types of characters, and that expression to be ill-typed.

This solves the Turkish problem, where the upper case I is two different charters in two different types (Turkish Roman script?), and the case folding functions likewise have disjoint types.

> I want Latin and Cyrillic to be different types of characters

How do you concatenate English and Ру́сская text, and what is the type of this sentence?

[Either [Latin] [Cyrillic]] is a very simply type taking advantage that the language only switches at word boundary.

Huh. That doesn't quite address my objection (CamelCase like EnglishEtРу́сская still un-works), but that's actually a good point in the overwhelming majority of cases. I'm not quite convinced this approach works in practice (I'm sticking with "A"="A"="A"), but I'd definitely like to see a more technically fleshed-out design.

Reply on news.ycombinator.com