Wait wait wait
Carl, you did not tell me that the errors we emit need to be correct!!
I think I need to go revisit some of my PRs...
859 karma · joined March 26, 2018
Carl, you did not tell me that the errors we emit need to be correct!!
I think I need to go revisit some of my PRs...
(It's also more difficult to support plugins effectively in a type checker like ty, than in a linter like ruff, since any non-trivial use case would likely require deep changes to how we represent types and how we implement type inference. That's not something that lends itself to a couple of simple hook extension points.)
https://github.com/python/cpython/blob/main/Parser/Python.as...
ty uses the same AST and parser as ruff. We don't use the ASDL grammar directly, because we store a few syntax nodes differently internally than how they're represented upstream. Our parser is hand-written in Rust. At first, our AST was also entirely hand-written, though we're moving in the direction of auto-generating more of it from a declarative grammar.
https://github.com/astral-sh/ruff/issues/15655
https://github.com/astral-sh/ruff/tree/main/crates/ruff_pyth...
https://github.com/astral-sh/ruff/blob/main/crates/ruff_pyth...
This is a known issue — we're currently defaulting to a conservative Python version, and `datetime.UTC` really doesn't exist until Python 3.11!
https://docs.python.org/3/library/datetime.html#datetime.UTC
We will probably change the default to "most recent supported Python version", but as mentioned elsewhere, this is very early and we're still working out these kinds of kinks!
This is true! We're also contributing salsa features upstream where we can, e.g. https://github.com/salsa-rs/salsa/pull/603
> The entire system is designed to be highly incremental so that it can eventually power a language server (e.g., only re-analyze affected files on code change).
From the thread:
> We haven't publicized it to-date, but all of this work has been happening in the open, in the Ruff repository.
We had conflated JS and TS support into a single release issue. JS support never landed, but TS did: https://github.blog/changelog/2024-03-14-precise-code-naviga...
When we first created the release issue, the thinking was that we wanted to launch support for those two languages at the same time, and for that support to be compatible with each other. (So we could correctly follow e.g. a TS library referencing a definition in an upstream JS library.)
Unfortunately we never got JS support to the point where we could GA it. Largely for scaling reasons — there is a HUGE amount of JS on GitHub, and JS is a dynamic enough language that we have to do more per-file processing on it than for TS or Python. We decided not to wait for JS to be GA-ready before releasing TS, but then never corrected the release issue to account for that.
Where we've found success is in stepping back and creating formalisms that are language-agnostic to begin with, and then using tree-sitter to manage the language-specific translations into that formalism. A good example is how we use stack graphs for precise code navigation [1], which uses a graph structure to encode a language's name binding semantics, and which we build up for several languages from tree-sitter parse trees [2].
[1] https://dcreager.net/talks/stack-graphs/
[2] https://github.com/github/stack-graphs/tree/main/languages
[1] https://lists.ozlabs.org/pipermail/ccan/2022-September/00141...
[1] https://docs.github.com/en/repositories/working-with-files/u...
We also have Precise Code Navigation — still currently only for Python [1], but we're working on other languages as fast as we can. As mentioned down-thread, that is built on our new stack graphs framework [2]. We're often asked why we're not leaning on something LSP-shaped for this. I go into a built of detail about that in a FOSDEM talk from a couple years back [3], and also in the Related Work section of a paper that we just published [4].
[1] https://docs.github.com/en/repositories/working-with-files/u...
[2] https://dcreager.net/talks/stack-graphs/
One often overlooked drawback of generating this data during CI is that you, the project owner, are now paying for the compute. Of course, maybe you qualify for a free tier. And if not, if you're already running a CI job for testing/linting/etc, the marginal cost of also generating code nav symbols might not be too bad. But one benefit of the stack graphs approach mentioned up-thread (and the code search indexing described in OP) is that all of the analysis and extraction work is done in dedicated server-side jobs that GitHub is footing the bill for.
On my team we have a weekly 30-minute meeting for non-work-related updates. A chance to talk about whatever you did in the past week outside of work, or about anything else that currently interests you. It typically prompts a lot of follow-on conversations in Slack afterwards.
It might be nicer if this happened organically, and didn't require an explicitly scheduled meeting. But we consider it important enough that we carve out the time to make it clear that it's a team norm.
[1] https://tree-sitter.github.io/tree-sitter/code-navigation-sy...
The build-based approach that you describe is also used by the Language Server Protocol (LSP) ecosystem. You've summarized the tradeoffs quite well! I've described a bit more about why we decided against a build-based/LSP approach here [4]. One of the biggest deciding factors is that at our scale, incremental processing is an absolute necessity, not a nice-to-have.
[1] https://github.blog/2021-12-09-introducing-stack-graphs/
[2] https://dcreager.net/talks/2021-strange-loop/
Another benefit to our approach is that it should be much easier to adapt OP's linter to work with other languages, since in a sense it's “parameterized” by the language grammar and queries. If you use Python's `ast` module, you can't easily adapt that code to work on Go programs, for instance.
The `tree-sitter generate` command also produces a full description of the grammar itself (typically written to src/grammar.json). In theory you could use this to generate each grammar implementation in other languages, as well. No one has tackled this yet, though, because in practice it has worked well enough to FFI in the C parser implementations.
Also, depending on what language you're trying to consume tree-sitter parsers in, we're trying to leverage the language's package ecosystem as much as we can to simplify this process. If you're using Rust, for instance, we publish crates for tree-sitter itself [1] and for various language grammars [2,3,4]. So you can often just add some dependencies to your Cargo.toml and the Rust build system takes care of everything for you. (Note that these crates also provide a more Rust-idiomatic API, instead of making you call the FFI'ed C functions directly.)
[1] https://crates.io/crates/tree-sitter
[2] https://crates.io/crates/tree-sitter-python