Hacker News Reader

Comment by schemescape | Hacker News Reader

Am I reading correctly that characters are 16 bits? If so, is this just for the boundaries (e.g. file system, web)?

In addition to what the other reply posted, Java has had compact strings for a while now[1], which uses 8-bit `byte` arrays for `String`s, and comes in handy for ISO-8859-1/Latin-1 strings to use a single byte per character.

[1] https://openjdk.org/jeps/254

One annoyance with that approach is that a single non-latin-1 codepoint is enough to double the size of the String; if it were an UTF-8 String, it would only add a couple of extra bytes.

At the expense of O(n) index access. At least Java and JS have compact strings. C# is always UTF-16 because they exposed too much implementation detail to be able to transparently do the compact string optimization.

> At the expense of O(n) index access.

Ah yes, that feature which means we must have 2x memory overhead and/or dynamic dispatch, yet is also totally useless in practice.

The fact of the matter is that there are millions of places with loops like for(...) {charAt(i)} so it is by definition useful in practice. If Java had pushed iterators more strongly, perhaps it would be possible to make the switch.

In all seriousness, what's the point of having the best JIT in the world if it can't reliably transform this antipattern into an iterator?

OpenJDK seems pretty averse to including special case optimizations, otherwise I'm sure we would have intrinsics at least for ArrayList/HashMap containing boxed primitives. AFAIK they are still noticeably slower than plain arrays or manually inlined implementations.

For the most basic case you wouldn't even need JIT support - just keep a cache field of the last index accessed and where it ended and decode based on that.

Java predates Unicode 3.1 (I think I got the version correct: but basically when Java was created, Unicode had less than 65536 codepoints). So Java had 16 bits chars from the get go and to this day is still backward compatible with earlier Java code.

You can also ask to get the "codepoint", which returns you an int.

But, yup, the Java char primitive is 16 bits.

It's a SNAFU hard to understate. SNAFU doesn't even being to describe it. But to be honest Unicode in itself is probably the biggest SNAFU of our entire field.

> But to be honest Unicode in itself is probably the biggest SNAFU of our entire field.

Explain.

EDIT: Oh! I construed this as a negative sentiment about Unicode, but perhaps it was just: "Yeah, it's complicated because language is complicated". Thanks, responders!

It was always going to be messy to create something that covered all of the writing systems in the world, but things like this didn't help, and is probably what they're referring to:

https://en.wikipedia.org/wiki/Han_unification

I can't explain what the previous poster was thinking, but Unicode is frustrating to work with. The problem is, that frustration just replicates the reality of the human writing systems Unicode captures. Nobody invented weird nonsense to make your life harder in Unicode, millions of humans developed a variety of weird nonsense to make their lives easier, over thousands of years and Unicode captures most of it.

Case for example, why the heck would there be exactly two versions of A, one of which usually only appears at the start of words, and not always? Other people's squiggles don't do that, but notably several European writing systems (Latin and Cyrillic) do. People who use those systems think ABCDE and abcde are somehow "similar" even though they don't seem very similar at all. Then they're surprised other people don't do that.

Not every human writing system has to be expressible in all contexts.

Popular languages on computers should generally work, though.

> Nobody invented weird nonsense to make your life harder in Unicode

The current state of emoji, unfortunately, makes this no longer true. (That being said, it's also designed in a way that's mostly ignorable if you don't need it. But only mostly.)

I don’t think that pictograms are weird nonsense. Symbology always had an important part in human history, and where else would you put it?

A markup language, or named entities, some length-prefixed blob, just generally something that doesn't require making UAX 29 more difficult.

I understand why emoji is as complicated as it is. I don't think it was a good idea to bake that complication into Unicode.

Emoji is not an additional complexity, it builds on top of the same primitives like Chinese characters and whatnot. Unless you claim supporting non-european languages is unnecessary, then emojis are a perfectly good “making use of” of unused regions.

> Emoji... builds on top of the same primitives like Chinese characters

It does not, and if you think it does it's obvious you didn't read UAX 29. GB11 is the most complex rule and only applies to emoji (well, and other Extended_Pictographics if you don't want to call the legacy pictographics emoji, but no Chinese).

Exactly, there's nothing really wrong with Unicode -- it's complicated, yes, but only because human scripts are.

It's the same with the windows API

Unicode is not a SNAFU. Though the UC has made mistakes here and there, the complexity of Unicode is only a mirror of the complexity of human scripts.

Please stop taking dumps on Unicode. It does nothing for actually getting it implemented and implemented well.

Seriously, the zoo of text encodings that reigned before widespread Unicode uptake was a pain the ass once you had data that went beyond 7 bits, especially for non-Roman scripts.

> the complexity of Unicode is only a mirror of the complexity of human scripts.

Oh please, Unicode didn't start out as an encoding covering human scripts - it was too late for that. It started out as an encoding to end all already existing encodings and that was a shit show with all kinds of weird and non printable symbols to begin with. It also didn't get better when Unicode started to add all kinds of shit like combining emojis into the mix.

> Unicode didn't start out as an encoding covering human scripts

> It started out as an encoding to end all already existing encodings

And those other encodings were encodings for a variety of human scripts.

Re: emojis, well, they're not that different from Kanjis, not really -- both are pictographs after all, one just has much more modern functionality (e.g., color).

Reply on news.ycombinator.com