Hacker News Reader

Comment by spaceywilly | Hacker News Reader

My question was, why was there not a unit test for this? It seems like it would be trivial to step through every character combination for each language they support and make sure it doesn't cause a crash.

View on HN

Unicode allows for 17 planes, each of 65,536 possible characters (or 'code points'). This gives a total of 1,114,112. This crashing sequence is five characters long.

Your unit tests would have to go through 1.71650179e30 sequences to be guaranteed to catch this one. At a test rate of 1 millisecond per sequence, that's just 4×10^9 × the age of the universe, according to wolfram alpha.

Rendering Indic scripts with AAT fonts involves a series of finite state machines that are stored in the individual font. So don't forget to multiply by the number of different fonts that each need to be tested.

> Unicode allows for 17 planes, each of 65,536 possible characters (or 'code points'). This gives a total of 1,114,112.

Allows for 17 planes, but only small portion of those are actually used. According to Wikipedia[1], currently Unicode has 148944 codepoints + 128k private use ones (which might, or might not make sense to include in unit tests). So your time estimate is off by mere 5 orders of magnitude.

(148944 ^ 5) milliseconds to years = 2.324 quadrillion years .

Doesn't really change the result, imho.

Do you know how was this bug found?

Are there just enough people using iOS that these sorts of bugs can be found by mistake, or is someone fuzzing CoreText? Perhaps that can be applied to provide some kind of test coverage? Even if it’s not complete?

This sequence begins the Telugu word for "knowledge" so maybe someone texted that to someone and it went viral from there. This is, of course, only speculation.

Does the word include the zwnj character? How do you input it?

It does not include the zwnj, that somehow snuck in. Most keyboards don't support directly inputting a zwnj, but may support it for specific combinations. For example my Marathi keyboard supports typing eyelash rephs (e.g. in र्‍क) which includes zwj.

However I'm not aware of any such things in Telugu aside from explicit virama-showing which rarely exists in input methods (and doesn't end up with zwnj in the position shown here, but that could have happened after editing).

Huge oversight on my part, the string starts with 0xC1C, 0xC4D, 0xC1E, 0xC3E i.e. without the ZWNJ. I'm stumped

https://en.wiktionary.org/wiki/%E0%B0%9C%E0%B1%8D%E0%B0%9E%E...

true, but characters could be mapped to equivalence categories according to logic in the code

I don't know if I agree with that. The sequence here is something like 5 or 6 characters. There's some tens of thousands of unicode characters, which suggests something like (10000)6 combinations to test. Fuzzing might be more fruitful, but even then I'm skeptical it can find obscure crashes. There's no replacement for defensive programming.

Concolic execution as a form of whitebox fuzzing is a very effective way to generate these kinds of crashes, especially if the crash is simple enough that it's the input stream that's generating it, and not positioning the program into a very specific state to get this kind of input string to crash it.

There are more possible character sequences than there are stars in the sky.

Why does everyone assume only this one specific character combination causes the issue? Generalists can't generalize, I guess. IOS/OSX are notoriously bad at unicode parsing... These have been problems for YEARS and there are many many known character combinations that cause these types of issues. You seriously think every character has unique code to process it? Of course not... there are general subsystems that process all of these things, and those can easily be tested. Literally, copy/pasting the Arabic word for "hummus" (حُمُّص) about 20 times will crash OSX/IOS. It's likely a mixture of RTL and LTR along w/ BOM and other unicode weirdness that causes these issues. That is something that is trivial to test for. For added effect, just shovel ح̷̵̷ُ̧̛̟͙̙̠͖̺̞̟̬͖͙̯̭̞̘͚̻̱͉̦̲̑ͫ͐ͬ̏̽ͨ̐̃̃́ͬ͑̃͛̍̃ͭ̄ͬ̊͗̆̇́͘͜͜͞

م̶̸̸ُّ̢̧̨͓̮̣̺̤̟͓͕̯̯̬͉͍̥̥̹͉͉̠̣̰̜̻͓͖̮̫͎̯͙͇̳͛ͩͨ͒ͦ̏̊͒ͩ̅̅̑ͤ̋ͮͩ̔̒͆̔̂͐ͧ͒̐ͭ̎̕̚͜͜͞͞͠ ص̑ͨͦ̆͗̌̔ͬͬ̈̌̑͏̧̙̰̩̭̜̮̺͚̼̙͉̱̭͉͖̤͞ concatenated a few times to any IOS or OSX application (chat apps work great), and you can crash even the most recent "fixed" versions of IOS/OSX, still, to this day, because these bugs have never been fixed at their core. IOS/OSX unicode support is awful, and will remain awful, if the past has been any indication of the future.

An article that claims to "pick apart" the issue, that only mentions "unicode" once in the article body is pretty sad... and no mentions of byte order mark, or endianness. Some dissection...

I don't say "unicode" because I'm not certain it has to do specifically with unicode handling, and not a font stack bug (that would occur on other encodings). One of the previous crashes, for example, had nothing to do with unicode -- it had to do with the fact that some arabic strings get larger, visually, when you shorten them. Folks these days say "unicode" when talking about anything relevant to non-Latin text, which munges up the issue, which is why I specifically avoiding saying that word.

BOM or endianness don't seem to be relevant to this bug.

The problem is not with the characters themselves though, it's with how CoreText processes and parses the unicode symbols.

Don't get me wrong, I know fonts can be malicious, even... but given the history[0] here, with these unicode[1] issues... I think it's pretty safe to say the issue is not with a specific font, per-se, but with CoreText and unicode parsing. For example, when I ran OSX on my old macbook pro, I used open source fonts on my terminal, and this unicode parsing bug still happened.

In 2015 an Apple spokesperson had this to say: "We are aware of an iMessage issue caused by a specific series of unicode characters and we will make a fix available in a software update."

> BOM or endianness don't seem to be relevant to this bug.

Yet on your blog you allude to just that with the left/right comments, though, to be fair, you state that you really don't know the problem:

> I don’t really have one guess as to what’s going on here – I’d love to see what people think – but my current guess is that the “affinity” of the virama to the left instead of the right confuses the algorithm that handles ZWNJs after viramas into thinking the ZWNJ applies to the virama (it doesn’t, there’s a consonant in between), and this leads to some numbers not matching up and causing a buffer overflow or something.

This is claimed to be a dissection of the issue, but there is not even a stack trace present, and yet you joke about that...

> Yes, I could attach a debugger to the crashing process and investigate that instead, but that’s no fun

Nah, you should do that... and you'll likely see that it's CoreText being the same old piece of shit as usual. If it was a font problem, then loading that font on a different system that uses a different rendering engine should reproduce the same problem. It doesn't, I tried that years ago.

Sure, I would posit, that potentially, this is a whole different bug, but... given the history, and the repeated failed attempts to fix this entire class of issues... it's safe to say that IOS and OSX do not handle unicode very well.

[0] https://www.theregister.co.uk/2013/09/04/unicode_of_death_cr...

[1] https://www.theregister.co.uk/2015/05/27/text_message_unicod...

I'm not talking about malicious fonts, but font stacks are complicated regardless of the encoding being fed in.

You seem to think I'm blaming it on the font. I'm not. I'm blaming it on the font stack (CoreText)

I avoided using the term "unicode" to refer to a bug in the font stack because font stack bugs don't always have to be specific to unicode.

You keep saying "unicode parsing"; that's a meaningless term in this context.

> Yet on your blog you allude to just that with the left/right comments, though, to be fair, you state that you really don't know the problem:

That doesn't have to do with endianness or BOM, that's a totally different kind of ordering. It's ordering of the code points, not the code units.

> This is claimed to be a dissection of the issue, but there is not even a stack trace present, and yet you joke about that...

It's a dissection of the string. Not a full debugging of the issue.

Googling, "If you tabulate all stars visible down to magnitude 6.5, thought to be the faintest stars still visible to the unaided eye, the entire sky contains some 9,000 stars. Since you can only see half the sky at any time, that means there are as many as 4,500 stars visible in your sky tonight."

Apparently not a very high bar.

I bet they have unit tests for all emojis...

this comment is under-rated. haha, you know it's true.... and sad.

Reply on news.ycombinator.com