Hacker News Reader

It'd be interesting to benchmark this on texts from different languages. English text will be very friendly to the branch predictor because it's all 1-byte codepoints. I think a language dominated by either 2-byte or 3-byte codepoints would be as well, mixing codepoint lengths is what would trip things up.

View on HN

Been a while, but I found it was best to assume all the text is ASCII, and fall back to a slower code path only if that turned out not to be the case.

A lot of tools that create JSON, XML, and so forth will escape any non ASCII characters by default. Python's json package, for example.

Agreed. I saw measurable speed increases when defaulting to ASCII (via macro) and only actually calling the decoder function if the value wasn't < 128.

#define cdpt(s) ((s[0] < 128) ? s[0] : cdpt_func(s))

https://github.com/torstenvl/trex/blob/master/src/trex.c

An all-ASCII fast-path is basically obligatory for anything that cares about performance. And then the "slow path" gets a guarantee that there will be varying-length data, and, as you'll want it to not actually be slow, you'll still need to avoid branching on the now-guaranteed-variable width.

Reply on news.ycombinator.com