Unicode Regular Expressions | Hacker News Reader

Unicode Regular Expressions

unicode.org

96 pointskawera26 comments

Nice, two pieces of technology which are always implemented incorrectly combined into a single standard.

This is not really a standard; nearly any behavior which is complicated is left undefined or expressed with vague, aspirational language ("an implementation should do this somehow, I dunno"). It doesn't look like regex implementers were consulted at any time.

“Should” likely should be interpreted as defined in https://www.ietf.org/rfc/rfc2119.txt.

I think there are a lot of should’s because the Unicode consortium realizes that there will be variability in feature sets of regular expression libraries. http://unicode.org/reports/tr18/#Conformance has quite a few shall’s to counteract that, but still leaves implementations to pick and choose features, as long as they document that.

HTML rendering of said RFC:

https://tools.ietf.org/html/rfc2119

“Should” has a very precise technical meaning in standards. Mentally it’s best to replace it with “MUST” (they are equivalent).

"Must" is equivalent to required, "should" is equivalent to recommended, so not at all the same.

True enough. I was conflating “should” with “shall” which is definitionally equal to “must”. The edit window has elapsed or I’d correct it.

True enough; however, when the behavior after the should isn't properly specified...

As a regex implementor, I am super appreciative of this document. I've consulted it quite a bit, and it helped guide me towards picking and choosing which Unicode features I should support and what their semantics should be. The upshot is that many other regex engines have either inspired this document or been inspired by it, which gives at least some consistency between implementations.

I also think it's clear from reading this document many times that it was very much informed by regex implementors. The text seems quite self-aware of the fact that some features may not be practical to implement in all scenarios.

It is certainly far from perfect, and I bet that if we got in a room together, we could vent about things in (or missing from) this document. But I am determined not to let perfection be the enemy of good, and this document has, at least in my experience, done much more good than harm.

Ditto to everything here. UTS #18 is a fantastic map for the implementor. I co-authored a paper about it: http://www.dfrws.org/sites/default/files/session-files/paper...

I still don't agree, although I respect both of your opinions as real regex implementors without a doubt.

It's just too complex a 'standard'. The amount of crap they had to 'retract' by the current version is telling, and features like 3.9 and 3.11 are speculative design by standards committee, which is the exact opposite of how these things should work. It's the ultimate unfunded mandate: "hooray, I'm writing a standard, why don't I shove some random crap in there that no-one implements now in existing libraries in the hope I get to set directions".

The standard should be minimal and well-defined. This is neither.

It's also preposterously self-important. Yeah, I'm so fascinated by the history of the Unicode standard I'm going to start introducing an "age" property so I can regex-match on when characters were entered. There's a certain attitude whereby these guys are just blithely shoveling junk into a standard that no-one needs. If you're desperate for all these exotic character properties put them in a godamn file as a character class and blat them in wherever you need them (yes, it would be nice if regex had macros).

Somewhere in all this nonsense is a good standard struggling to get out. They need to shit or get off the pot. Anything that they don't know how to do or that no-one knows how to do should be completely deleted. Anything that they offer 15 different choices as to how to do ... they need to figure out which is the best one and pick that. It's a standards documents, it's supposed to make choices, it's not a potted plant.

It has a bad case of featuritis; it reads both discursively and self-consciously, like it knows that there's really no justification for using regexps to specify named properties, but it just can't help itself.

Some of the Level 3 features venture too far afield from Unicode, too, e.g., 3.11 is "implement YACC" and has nothing to do with Unicode and 3.7 is "implement a streaming regexp engine, we dare you" (challenge accepted, Unicode dweebs!).

So, it is a horrible standard by which to judge overall conformance, but no one really does that. I think the value is in lighting the way through a broad array of thorny Unicode topics. And I think that the implementor who grapples with a majority subset of the sensible topics will wind up with an engine that handles users' Unicode needs.

Since you mentioned being a regex implementor, could you give me your opinion of the PCRE2 library? I had always assumed it to be a well polished mature library (simply because it's popular), but learnt this week that it doesn't support even basic Unicode properties like `\p{Alphabetic}`. According to the linked article, it wouldn't pass Level 1.2 Unicode support.

A lot of code points used in Indic languages are outside the `L` categories, and not having the Alphabetic property (and having both `\w` and the POSIX `:alpha:` match only within `L`) is a significant annoyance when parsing mixed-language text (in Julia, which uses PCRE2 as the backend for regex). Is PCRE2 particularly behind in Unicode property implementation, or is this par for the course in common regex engines today?

At a high level, PCRE2 is one of the most mature and fastest regex implementations out there (among ones that are open and that I know about).

w.r.t. to Unicode support, it is fairly par for the course. There are some regex implementations that have very good Unicode support. Python's `regex` library (not `re` in the standard library) is very comprehensive with Unicode, and I think the regex library that `icgrep` uses is also fairly comprehensive. Both of those extend well into level 2 of UTS#18. Rust's regex library (the one I wrote) has fairly complete level 1 support with a few hiccups, but that is a somewhat recent development.

I imagine ICU's regex engine also has fairly complete UTS#18 support as well, although I've never used it myself. Similarly for Oniguruma.

icgrep is the only one of the above that I mention that doesn't have a reputation for being slow (and is actually quite fast). But... its toolchain dependencies are considerable.

PCRE2's Unicode support is pretty good, but definitely has its quirks. It definitely appears to support a limited set of properties, and appears mostly restricted to general categories and scripts, which gives a fair bit of breadth. You might be able to get close to \p{Alphabetic} by building its constituent parts: https://unicode.org/reports/tr44/#Alphabetic --- It looks like PCRE2 supports everything in Lowercase + Uppercase + Lt + Lm + Lo + Nl + Other_Alphabetic except for `Other_Alphabetic` (this is according to `man pcre2pattern`). Unfortunately, Other_Alphabetic is quite large, so it might be hard to just handjam that easily.

Interestingly, with all that said, PCRE2 does support a Unicode interpretation of \w, which is defined by Unicode to be: http://unicode.org/reports/tr18/#word

    \p{alpha}
    \p{gc=Mark}
    \p{digit}
    \p{gc=Connector_Punctuation}
    \p{Join_Control}

which is a superset of \p{Alphabetic} (\p{alpha} is an alias for \p{alphabetic}). Note though that \w by default in PCRE2 is ASCII-only. You need to enable the UCP flag to get Unicode support.

You mention Julia, and they've definitely had issues with PCRE2. For example, one thing that Rust's regex engine supports that PCRE2 doesn't is the ability to have Unicode features enabled while simultaneously searching possibly invalid UTF-8 without additional costly checks or undefined behavior. PCRE2 can't do that. See: https://lists.exim.org/lurker/message/20180412.091038.080cf6... and https://github.com/JuliaLang/julia/pull/26731

If you want to work with these in Python, use the "regex" package. [1] Not to be confused with the "re" module in the standard library.

Also, maybe there should be a (2016) or even a (2004) in the title! (2016 is when this document was last revised, and 2004 is when it became a Unicode standard.)

[1] https://pypi.org/project/regex/

Wasn't "regex" also the name of one of re's predecessors?

Yes. It was deprecated in 1.5(!) and finally removed in 2.5:

https://docs.python.org/release/1.5.1/lib/module-regex.html

https://docs.python.org/2/whatsnew/2.5.html#new-improved-and...

I don't think the regex package implements this. At least the hex notation is not supported:

  >>> import regex
  >>> regex.compile(r'\u{3040}')
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib/python3/dist-packages/regex.py", line 345, in compile
      return _compile(pattern, flags, kwargs)
    File "/usr/lib/python3/dist-packages/regex.py", line 507, in _compile
      caught_exception.pos)
  _regex_core.error: incomplete escape \u at position 2

Hmm, okay. It seems to just use Python's unicode escape syntax, instead of what the standard says. But the package does support things like the complex character classes, and identifying word boundaries.

(The algorithm for identifying word boundaries in most languages without ASCII/English assumptions is quite complex and useful, as I can say from having tried to reinvent half of it before learning about the standard and the regex package. It's not a panacea -- it won't do anything useful with languages where word boundaries require lexical knowledge, like Chinese, Japanese, and Thai -- but other than that it handles all the edge cases you never would have thought of.)

Yeah, basically, the standard is a little sneaky on this point. It doesn't actually require a specific syntax, but rather, simply that being able to write Unicode codepoints in hexadecimal representation is possible. Notice that it says, "... shall supply a mechanism ..." rather than "this mechanism."

Of course, it's probably a good idea to follow the sample syntax provided. :-)

Hearing "Unicode Regular Expressions" makes me think of some other "fantastic" technology combinations:

- XML configuration for Windows DLL dependency

- CORBA aware DST Library

> - XML configuration for Windows DLL dependency

Oh boy! You're in luck today, you'll learn about side-by-side assemblies and application manifests: https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

This will make a difference how we write regexes. Look at the level 1 properties:

   [\p{letter} \p{decimal number}]

And compare for instance to the POSIX regexp character classes (a reference: https://en.wikibooks.org/wiki/Regular_Expressions/POSIX_Basi...)

  [[:alpha:] [:digit:]]

Given that the Unicode standard goes much beyond, proper implementations should help to write regexps for the modern web, encountering problems we are typically faced with. It's a pity the standard goes not into detail there, thought (for instance, similiarity of characters).

> This will make a difference how we write regexes. Look at the level 1 properties:

> [\p{letter} \p{decimal number}]

> And compare for instance to the POSIX regexp character classes

> [[:alpha:] [:digit:]]

No need for the future tense; Perl's had that at least since v5.12: http://perldoc.perl.org/5.12.0/perldelta.html#Unicode-overha... .

Is there a limit to how complicated and ornate a character can be, besides pixels. It's amazing Chinese characters can be rendered so well across many browsers and websites

It depends on the font format. If TrueType doesn't let you code something, you could use an SVG font etc. etc. Almost all fonts in the wild are outline/scalable, so that imposes some limitations.

Reply on news.ycombinator.com