Despite the worrying talk of tokenization, it looks scannerless. Yay.
One big grammar. JSX is blended in. And ecmascript versions aren't segregated. Which I didn't expect... but the motivation wasn't writing picky compilers.
[1] https://github.com/tree-sitter/tree-sitter-javascript/blob/m...
We have to handle JSX and all language versions combined because for our use case, users need to be able to open up `.js` files and have them Just Work.
It's a similar story with Python and Ruby - we have to handle the union of all language versions.
Nod. A single big grammar just wasn't what I'd thoughtlessly expected. When I did something similar, there was a gaggle of small grammars slicing up javascript aspects and history, then variously composed into assorted language grammar variants. But part of the motivation for that was sharing grammar fragments across languages, and "parse exactly language version X" in support of parser validation and compiler use. So a bit different use cases.
Any aspects of the 'not' part of 'mostly' which one should bear in mind?
In most cases, you don't have to think about it; the obvious way to write something is the right way. There are cases where its helpful to have a mental model of how lexing works.
Nifty. So explicit GLR forks at rule level (`conflicts:`), non-forking token "conflict" resolution[1], no token regex backtracking pressure across tokens, and token resolution at a single code position can(?) differ across conflicting rules? I'm uncertain on that last bit.
> In most cases, you don't have to think about it
Well yes, but, some of us crave much more syntactically flexible languages. :)
[1] https://tree-sitter.github.io/tree-sitter/creating-parsers#c...