Comment by masklinn | Hacker News Reader

Comment by masklinn | Hacker News Reader

Parent Full thread

> What is the practicality of an unbounded number of possible graphemes?

That question doesn't really make sense. Unbounded cluster sizes is simply a result of unicode design because there's no reason to bound it.

> Does anyone in real writing of languages and not some concocted test case need this?

In the real writing of languages there are languages with grapheme clusters of size 4+ e.g. क्षि (Devanagari kshi) is a single cluster made up of क, ्, ष and ि.

> I understood there is some conflict in the encoding of kanji into unicode, does this relate?

It's unrelated. Han Unification compressed the number of clusters by unifying han characters across simplified Chinese, traditional Chinese, Japanese and Korean. Han characters were an issue in the original 16 bits unicode because supporting all of them was estimated as possibly topping out 100,000 codepoints (where 16 bits unicode only supported 65k codepoints). AFAIK Han characters don't use much if any composition (outside of romanisation). Brahmic scripts, Hebrew and Arabic scripts, on the other hand, pile on diacritics.

    >> What is the practicality of an unbounded number of   
    >>  possible graphemes?

    >That question doesn't really make sense. Unbounded  
    >cluster sizes is simply a result of unicode design 
    >because there's no reason to bound it.

The standard itself doesn't mention any bounds but there is Unicode Standard Annex #15 - Unicode Normalization Forms which defines the Stream-Safe Text Format.

    UAX15-D3. Stream-Safe Text Format: A Unicode string is said to be in             
    Stream-Safe Text Format if it would not contain any sequences of
    non-starters longer than 30 characters in length when normalized
    to NFKD.

    * Such a string can be normalized in buffered serialization with a
      buffer size of 32 characters, which would require no more than 
      128 bytes in any Unicode Encoding Form.
                                                                                     
    * Incorrect buffer handling can introduce subtle errors in the
      results. Any buffered implementation should be carefully checked
      against the normalization test data.
                                                                                     
    * The value of 30 is chosen to be significantly beyond what is
      required for any linguistic or technical usage. While it would have              
      been feasible to chose a smaller number, this value provides a very              
      wide margin, yet is well within the buffer size limits of practical              
      implementations.
                                                                                     
    * NFKD was chosen for the definition because it produces the
      potentially longest sequences of non-starters from the same text.

Funny thing I noticed when looking this up: The The Unicode Standard Version 7.0.0 PDF has exactly 1000 pages.

[1] http://www.unicode.org/reports/tr15/#Stream_Safe_Text_Format

> That question doesn't really make sense. Unbounded cluster sizes is simply a result of unicode design because there's no reason to bound it.

If I'm trying to draw a character on the screen, it makes a lot of sense. I guess I'm struck about how hard it would be to select the 4th through 8th character displayed on the screen from rows 3 through 13.

Displaying characters on the screen, and giving the ability to select text, is indeed quite a difficult and complex question in the face of full support of the world's written languages.

Glyphs vary in width. There are combining characters. There are ligatures (and in some languages, ligatures are necessary for the text to be readable, not just stylistic). There is contextual reordering of characters. There's bi-directional text, which can switch direction mid-sentence when quoting some Roman text in Hebrew or vice versa. There's vertical text. There are a number of fonts, font weights, font styles, font sizes, font features like alternative characters, language dependent selection of glyphs, and so on, plus fallback if the user doesn't have the designated fonts or fonts that cover the given Unicode range installed. There's line breaking, which can't always operate on spaces because not all languages include spaces between words. There's justification of text, distributing the spacing between characters to make a block of text even on both sides.

All of this complexity exists regardless of the encoding that you use. Simply picking a fixed width encoding will not do anything about the vast majority of the complexity; and the fixed width encoding will cause problems of its own (size bloat of strings in memory), as well as leading people astray in thinking that maybe they can get away with making some of the same assumptions that apply to ASCII on a fixed-width terminal (and even there, assuming that 1 character equals 1 unit of width fails in the case of tabs), when in fact they can't and most text handling beyond simply appending strings together is best handled by dedicated libraries that have had hundreds of developer-years of effort put into solving all of these problems.

Reply on news.ycombinator.com