Hacker News Reader

Comment by 4ad | Hacker News Reader

The Plan9 grep deals with Unicode correctly and is very, very fast. Claiming that

> dealing with Unicode correctly is a very expensive business

is plain FUD.

UTF-8 has zero overhead in this case (not near zero, but actual zero overhead). Grep doesn't even need to be aware of UTF-8. It doesn't need to be aware where a rune starts and ends in the byte stream. UTF-8 is designed in such a way that matching a regexp only requires byte comparisons, exactly as for ASCII.

View on HN

Unfortunately, it's not as simple as that. There is more than one way of representing many text sequences using Unicode; for example, combining diacritics vs accented characters. To do the Right Thing (tm), you need to normalize, and that adds at least another pass over the whole string.

If you just deal with it on a byte by byte basis, you're not actually Unicode compatible. It might be fast, but it's not Correct.

In addition, conforming Unicode implementations have to ensure that their input is well-formed Unicode, and reject non-conforming characters (for some security-related reasons). So the original goal of UTF8, that it could be usable for straight byte-level search (the UTF8 RFC mentions a goal of allowing Boyer-Moore to be applied directly) is no longer really possible.

edit: a discussion on this subject from a few months ago, where I was corrected on that point: http://news.ycombinator.com/item?id=2860932

It's not necessarily grep's responsibility to normalize strings. They could very well stipulate that it is the responsibility of the user to make sure the both the input and the search expression are already normalized. That way, the normalization could be done by a separate program, resulting in the reusability and separation of concerns that is The Unix Way.

I fully agree, if you redefine grep as a byte search program, rather than for text search. Rule it out for most languages apart from English.

To have any usable grep with applicability beyond programming languages, English-language log-files, etc. (i.e. massive Anglocentrism), the normalizing would need to be built-in. Sketched:

    #!/usr/bin/bash
    regex="$1"; shift
    unicode-normalize-to-utf8 "$@" | byte-grep "$(utf8-byteify-regex "$regex")"

> The Plan9 grep deals with Unicode correctly and is very, very fast. Claiming that

>> dealing with Unicode correctly is a very expensive business

> is plain FUD.

If you think this is true you don't know unicode enough. I don't know how plan9 grep is implemented but if it does it with near zero overhead then it's not actually supporting Unicode, but only with a small subset of Unicode that they like. I also probably agree that that's the best part of Unicode and the rest of Unicode could burn in a fire but it's still not Unicode.

> If you think this is true you don't know unicode enough.

Yes, well, that is sort of a given for any statement about unicode (including this one).

You are conflating UTF with Unicode.

Reply on news.ycombinator.com