Against Query Based Compilers
matklad.github.io
matklad.github.io
Why aren't you using an IDE with "navigate to definition" conveniently bound to something like middle-click or ctrl-click?
I haven't used a language/IDE combination without this feature in decades.
This is particularly true with cpp projects, where wrestling with cmake and co just isn’t how i want to spend my time to answer a question about why something is slow or what a serialized object shape should be under various circumstances.
Every time I've heard someone say a version of this, invariably they've spent more time doing things manually than the properly mechanised method would have achieved.
There's a seductive immediacy to certain quick & dirty manual processes that in the sum are slower if measured with a stopwatch. It could be as trivial as not having syntax highlighting means that you miss an issue because you didn't notice that something was commented out or a quoted string was quoted incorrectly.
Similarly, I've argued with many developers a few decades back that insisted that the fancy keyboard bindings of EMACS or Vim made them faster, but if raced against someone with an IDE they lost badly every time, no exceptions.
There’s all sorts of workflows where vim would mog the IDE workflow you’re really excited about, like pressing E in lazy git to make a quick tweak to a diff. Or ctrl-G in claude code.
I wouldn’t be so sure you’ve cracked the code on the best workflow that has no negative trade offs. Everyone thinks that about their workflow until they use it long enough to see where it snags.
... but you do more often that the quick & dirty approach really allows.
I was just watching the Veritasium episode on the XZ tools hack, which was in part caused by poor tooling.
The attacker purposefully obfuscated his change, making a bunch of "non-changes" such as rearranging whitespace and comments to hide the fact that he didn't actually change the C code to "fix" the bug in the binary blob that contained the malware payload.
You will miss things like this without the proper tooling.
I use IDEs in a large part because they have dramatically better diff tools than CLI tools or even GitHub.
> you’ve cracked the code on the best workflow
I would argue that the ideal tooling doesn't even exist yet, which is why I don't believe that I've got the best possible setup nailed. Not yet.
My main argument is this:
Between each keypress in a "fancy text editor" of any flavour, an ordinary CPU could have processed something like 10 billion instructions. If you spend even a minute staring at the screen, you're "wasting" trillions of possible things the computer could be doing to help you.
Throw a GPU into the mix and the waste becomes absurd.
There's an awful lot the computer could be doing to help developers avoid mistakes, make their code more secure, analyse the consequences of each tiny change, etc...
It's very hard to explain without writing something the length of War & Peace, so let me leave you with a real world example of what I mean from a related field:
There's two kinds of firewall GUIs.
One kind shows you the real-time "hit rate" of each rule, showing packets and bytes matched, or whatever.
The other kind doesn't.
One kind dramatically reduces "oops" errors.
The other kind doesn't. It's the most common type however, because it's much easier to develop as a product. It's the lazy thing. It's the product broken down into independent teams doing their own thing: the "config team" doing their thing and the "metrics" team doing theirs, no overlap. It's Conway's law.
IDEs shouldn't be fancy text editors. They should be constantly analysing the code to death, with AIs, proof assistants, virtual machines, instrumentation, whatever. Bits and pieces of this exist now, scattered, incomplete, and requiring manual setup.
One day we'll have these seamlessly integrated into a cohesive whole, and you'd be nuts to use anything else.
... one day.
When IDEs do resolve this it tends to be because they built some index to look up these identifiers, which is likely taking up a portion of your memory. A language that statically tells you with an identifier comes from will take out less resources, and your IDE can collapse the import anyways.
So not sure why you feel so strongly about a language design whose ambiguity necessitates consuming additional resources to show you your little drop-down menu.
Is it really "at a glance"?
In most languages either by convention or requirement the "import" or "using" statements are collected at the beginning of a file. Once you've scrolled down even a few lines, the context is gone.
Also, determining what exactly is bound where is decidedly non-trivial in many languages due to keywords such as "var" and "let", overloaded function/method definitions, etc...
Sure, a human can do this with 95% or better accuracy, but that 5% can be a killer during a complex troubleshooting session if you guess wrong.
That's why I strongly prefer IDEs and having a purely mechanical process to resolve the dependencies so I can know exactly what things are instead of hoping my intuitions were correct.
The problem I'm trying to solve is more around ensuring that purely additive changes in libraries aren't technically breaking due to the risk of name clashes than general discoverability though.
> In contrast, you can’t really parse a file in Rust...Expanding macros requires name resolution, which, in Rust, is a crate-wide, rather than a file-wide operation...Similarly, the nature of the trait system is such that impl blocks relevant to a particular method call can be found almost anywhere...
matklad doesn't even mention dynamic languages, where perfect name resolution is undecidable. "Fast static analysis, easy static analysis, language designed for static analysis - pick two".
Rust's IDE integration is fast and deep, and I've heard TypeScript's is too, so "easy static analysis" may not be important today. I believe it will as coding evolves due to LLMs, albeit without evidence and I'm not quite sure how.
Where does this come from and what's the explanation?
- You can write a static analysis that is fast and straightforward if you design the language around it (fast + easy + !language)
- Otherwise, you’ll run into features that prevent that. Ex (stolen from an above comment): name resolution within unqualified imports
- For many such features, you can write a brute-force solution. Ex: scan all possible definitions within the unqualified import until you find the target name (!fast + easy + language)
- Or you can optimize this solution, using incremental cached lookup tables (fast + !easy + language)
Of course, there are languages with features that make most static analyses for them neither fast nor easy (ex: for name resolution, eval or macros). But it holds in many cases, ex:
- Parsing: basic syntax (!language), GLR parser or LL parser with big alternatives (!fast), IELR parser or LL parser with smart alternatives (!easy)
- Type inference: many type annotations (!language), brute-force enumerate candidates (!fast), Hindley-Milner or bidirectional, possibly extended for features like trait resolution (!easy)
- Allocation: manual (!language), put everything on the heap (!fast), heuristics, implicit boxing and unboxing (!easy)
There’s a trend (I also don’t know where it originates) that most “data processing” problems can be solved with an easier slower or trickier faster algorithm (hence, competitive programming). Static analyses are this class of problem, but for them (sometimes) there is a third option: simplify the problem by putting hints in the language.
Rust-analyzer reimplemented the frontend in a more IDE-friendly architecture, but focused more on name resolution than on type checking. So it delegated diagnostics to literally just running `cargo check`.
As parts of rustc get rewritten over time (the trait solver, borrow checker) they have also been made more IDE-friendly and reusable, so rust-analyzer is slowly gaining the ability to surface more type checking diagnostics as you edit, without delegating to `cargo check`.
I think you might have meant "language *not* designed for static analysis".
The specific example is bogus.
Merkle trees and their many variants exist to solve precisely this problem.
For more compiler-specific scenarios there exist vaguely similar solutions to the issues introduced by incremental compilation such as splitting up monolithic executables into many small dynamically loaded libraries (only during development time), or taking that to the extreme, hot code reloading at the function level.
> ... only because Zig language is carefully designed to allow this.
Is the key point. Rust wasn't designed for incremental compilation, and most legacy languages like C only allow it in a useful way because they were designed in the era "kilobytes" of system memory and hence they're very frugal in their design.
Other than Zig, the only modern language I'm aware of that was "co-designed" for efficient compilation and incremental reload is Jai.
> Expanding macros requires name resolution, which, in Rust, is a crate-wide, rather than a file-wide operation.
Sure, but macros change infrequently, so that ought to be a highly cacheable pure function for most edits.
> The above scheme works only if the language has a property that changing the body of function foo (not touching its signature) can’t introduce type errors into an unrelated function bar.
AFAIK Rust and most other statically typed languages have this property. Optimisations such as inlining can mess with the cacheability of code-gen in all languages.
Actually, auto traits thwart this in some places for Rust: https://rust-lang.github.io/impl-trait-initiative/explainer/....
And they were created to minimize writing down what auto traits your types implemented. You can filter by them, but they are assumed automatic bounds.
And interestingly enough Zig doesn't have this property either it leaks constness, alignment and similar info.
The statement was about introducing type errors, not cacheability of code-gen.
Even more, I can't imagine how people write language servers with full reprocessing on each document change (even with query-based approach). How do they deal with broken syntax? isn't it generally impossible to recovery syntactic errors, so that the end result is as good as expected?
> isn't it generally impossible to recovery syntactic errors
AFAIK Zig was designed in such way that, in most cases, you can parse each line independently, like their strings by default are single line, you need a special syntax for multiline strings that still are per line basis.
> AFAIK Zig was designed in such way that, in most cases, you can parse each line independently
It can't be true. Basic stuff like opening/closing braces introduces some global syntax state, so that line-by-line parsing isn't possible. Do you mean something like tokenization instead? It is in many cases possible. But for true LSP it's almost useless.
JetBrains (not an LSP) works up to the syntax error, then recovers and works after. In some cases that are not hard to trigger (e.g. after a closing delimiter), the recovery is wrong and everything after is messed up. But in practice that’s rarely an issue; if I want those later diagnostics and code actions, I fix the delimiters, and the analysis updates quickly.
The last clause is a great point and one I will try to remember.
query based compilers are equivalent to component UI frameworks / signals / self-adjusting computation / spreadsheets. signia signals library implements this idea of using the previous output + diffs to compute the next output (https://signia.tldraw.dev/docs/incremental#using-diffs-in-co...)
Rust needs macros because the language is extremely strict and verbose. Macros are the main escape hatch, allowing you to simplify your code while maintaining memory safety without a GC.
Zig doesn't have memory safety as a goal so it seems like an unfair comparison.
fn pin(x: T) -> Pin<&mut T> { ... }
I won't respond further.
> It would be perfectly possible to design a variant of Rust that gets you to 80-90% of Rust's usability, with the same safety, without macros.
i then present an api that i think relies on macros to expose a safe api
> Of course macros are good for encapsulation and abstraction, but that's a different subject.
no it's not. exposing safe abstractions is pretty much rust's raison d'être
The "comparison" (it isn't one--it's a technical point; "fairness" is not remotely relevant--Rust isn't being attacked and doesn't need defending) is between a language with macros and a language without macros--why the languages have or don't have macros isn't relevant.
Zig is strict and verbose and could benefit from macros but for numerous reasons doesn't have them.
Zig also has no GC and does have memory safety as a goal (but not the primary one, and not provable memory safety)--but none of this is at all relevant to the point the OP made, which is strictly about one language having macros and the other language not having them.