Comment by ygra | Hacker News Reader

Comment by ygra | Hacker News Reader

Parent Full thread

There are multiple factors at play here and it ultimately comes down to what you need:

If you to access code points, then UTF-32 gives you a nice fixed-width (albeit quite large) encoding to work with. Some applications or libraries may use UTF-32 internally to speed things up.

If you deal with things on screen or what humans perceive as character (graphemes) then there is nothing really that can help you at the encoding level because graphemes can be arbitrarily long with regard to code points: e̬̱ͤ̈́̿̂̅́͟x̰̞̻̼̻̼͉̰ͫͧ̑͂͊͜ͅa̶̳͖͔̞̰͈̯͓͆̓̎̉̏̎̚m̛̮̩͙ͦ͗͆̄̋́̄p͎̥̠͈̮͚̾ͅl̛̗̈̄e̴̞̫͖̝̘̋̑̄̎ͥͬ̾ (this probably doesn't even render properly).

Sadly, most APIs use either UTF-8 or UTF-16 and consider code units the most important entity, not code points. And that's inexcusable and frankly a mess, because I can count the number of times I needed to access a code unit instead of a code point on zero hands.

> If you deal with things on screen or what humans perceive as character (graphemes) then there is nothing really that can help you at the encoding level because graphemes can be arbitrarily long with regard to code points:

> e̬̱ͤ̈́̿̂̅́͟x̰̞̻̼̻̼͉̰ͫͧ̑͂͊͜ͅa̶̳͖͔̞̰͈̯͓͆̓̎̉̏̎̚m̛̮̩͙ͦ͗͆̄̋́̄p͎̥̠͈̮͚̾ͅl̛̗̈̄e̴̞̫͖̝̘̋̑̄̎ͥͬ̾

As how many characters am I supposed to perceive this, anyway? Some of those tiny squiggles are in fact letters (I mean they look like letters, I see an e,m,u,o,i and r in there). What if you'd make a crossword this way?

It says "example" with a pile of combining characters (77 to be precise): http://i.imgur.com/acyHBkR.png

I've experienced crosswords where e.g. the word going across uses a diacritic on a particular letter ("élan") and the word going down doesn't. It drives me crazy.

FYI, that is the way crosswords work in italian, accents are considered irrelevant.

For index based operations code units are more useful than code points.

a = s.indexOf("e̬̱ͤ̈́̿̂̅́͟")

b = s.indexOf("m̛̮̩͙ͦ͗͆̄̋́̄")

s[a:b]

This operation is fast with code unit indices and slow with code point indices.

If you use stream-safe normalization rules, then you have a worst-case limit of 30 combining code points in a row. You could even make an array of 128 byte single-character-strings if you wanted.

Please define what you mean by code point and code unit.

I mean the same thing that Unicode uses these terms for.

Code Unit. The minimal bit combination that can represent a unit of encoded text for processing or interchange. The Unicode Standard uses 8-bit code units in the UTF-8 encoding form, 16-bit code units in the UTF-16 encoding form, and 32-bit code units in the UTF-32 encoding form.

Code Point. (1) Any value in the Unicode codespace; that is, the range of integers from 0 to 10FFFF16. (See definition D10 in Section 3.4, Characters and Encoding.) Not all code points are assigned to encoded characters. See code point type. (2) A value, or position, for a character, in any coded character set.

From http://www.unicode.org/glossary/

Reply on news.ycombinator.com