Yes, a better Perl script [1] :-)
In C, re2c [2] provides something similar, and there are similar tools available in other languages.
[1] https://metacpan.org/pod/Devel::Tokenizer::C
[2] http://re2c.org/
In C, re2c [2] provides something similar, and there are similar tools available in other languages.
[1] https://metacpan.org/pod/Devel::Tokenizer::C
[2] http://re2c.org/
Haven't checked re2c but it looks like a naive lexer implementation generating a static DFA bytewise, not optimized for speed nor memory.
Better generators should really operate on the wordlevel or wider (SSE), not bytes. These are ~8x faster, in some case up to 200x. http://blogs.perl.org/users/rurban/2014/08/perfect-hashes-an...