Hi! I took a shot at vectorized UTF-8 validation in 2012. I just put this faster validator, and Daniel Lemire's code, into my test harness at https://github.com/andrewffff/utf8fuzz/tree/2019_compare

On an i7-7800X (clang 6.0.0-1ubuntu2, WSL, don't trust my numbers) my benchmarks showed about the same relationship between SSE4, AVX2 and Lemire's code, as your benchmarks did. My own attempt is about half as fast. https://raw.githubusercontent.com/andrewffff/utf8fuzz/2019_c...

A few examples of invalid UTF-8 from Markus Kuhn's suite pass this validator right now, specifically 4.1.3, 4.2.3 and 4.3.3. My randomized tests, which compare the results from different validators, also fail a small percentage of the time, I'd guess for the same reason. https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt

I'm really interested in the different approaches taken here. Fortunately both Daniel and you've communicated what you were doing, I think it's going to take longer for me to re-comprehend my own approach!

Thanks for sharing this.