Comment by matklad | Hacker News Reader

Comment by matklad | Hacker News Reader

Parent Full thread

This depends on the compilation model in use. If that’s a traditional pipeline of phases, where the result is an AST data structure for the whole CU which gets annotated with types, then, yes, “enqueue new analysis” makes sense.

If the compilation model lazy & query based, then theres’s just no “enqueue a subsequent parse”, but rather “give me the type on this thing doing as little analysis as possible”. You got to re-use symbols and declarations not because you heuristicsly adjust offsets and assume they are otherwise valid, but because underlying analysis reasons, precisely, they they are valid and re-usable.

The two models are quite different at the core, and aren’t really an evolution of one into another.

Which of the two approaches to use, and, consequently, whether to think about cancellation at all, depends heavily on the language in question. If the language allows for lazy analysis, it’s probably a better bet, as that gives you correct results faster.

I only know a little bit about zig, but I would have assumed that the latter mechanism you are describing isn't really possible. Most languages with "advanced" features and type systems really need a full semantic tree to do anything meaningful due to the complexities of compile time code and type instantiation.

Even if we are using this latter model, the user is only editing one file at a time. What if the completion source is in a different file altogether? Generally, you're going to be indexing the entire codebase anyways, so we're splitting hairs over a potential optimization in just one facet of the indexer.

> Even if we are using this latter model, the user is only editing one file at a time.

I'm not necessarily arguing against your bigger point but this is very frequently a wrong assumption and strikes me as designing around an idealized view of the problem that you would like to be the case, not actual reality. Code generation scripts, formatters, auto-fixers and the like can modify many files and many parts of those files "at a time", unless you have a pedantic view of what "at a time" means. Almost all LSPs fail in these modes of operation and disallow external tools from participating in the code base, which is very annoying and makes them less useful.

Having to actually (re-)open a file so the LSP can "see" changes made to it by an external program is not something that should ever be needed but happens a lot with some of the most worked-on language servers (TypeScript comes to mind).

According to the spec it's up to your IDE to notify the LSP when files change (even due to an external software) then the LSP reads the new versions.

Typescript also doesn't use an LSP as it predates the introduction of LSPs.

What does it mean for a language "not to use an LSP"? The most straightforward way I can think to interpret that statement is that it's saying that a TypeScript language server doesn't exist at all, which certainly doesn't seem to be the case based on finding https://github.com/typescript-language-server as the top result when googling "typescript language server". I guess you might be saying that it's not a first-party language server, but I'm not really sure why that would be relevant when discussing properties of language server implementations; the implementation clearly exists, and even if very few people are using it, that doesn't mean GP's point about a flaw in its design wouldn't still apply.

I meant both VS Code and VS use tsserver which doesn't follow the LSP protocol. Yes there is an LSP for other editors.

I don't see the flaw in the design, it's more likely the editor isn't behaving according to spec.

Either way, all those files need to be marked dirty and reindexed, so I suppose I'm just not sure how the topic at hand is relevant. Is the proposal that we attempt to treat each dirtied file as though incremental edits were imminent? Because this is precisely the wrong assumption you're raising. Ultimately, if you need to reopen a file in order to see changes on disk reflected, that's a bug with your lsp server, nothing more.

> Either way, all those files need to be marked dirty and reindexed, so I suppose I'm just not sure how the topic at hand is relevant.

You stated an assumption as part of your argument (only one file is being edited at a time) and that assumption is very often false. If that assumption doesn't matter I don't know why you brought it up in the first place. I stated pretty clearly that I wasn't arguing against your larger point but the assumption you explicitly stated was false.

> Ultimately, if you need to reopen a file in order to see changes on disk reflected, that's a bug with your lsp server, nothing more.

You don't say? It's the kind of misbehavior you get when people make assumptions that are false and bake them into the design of things.

And that assumption is relevant in the context of OPs concern but not yours. You created a different hypothetical and orthogonal situation where the assumption need not apply, and to be honest, I lost the plot a bit. It's possible for there to be a branching set of concerns, each of which having their own set of perfectly valid but otherwise disjoint assumptions.

> Most languages with "advanced" features and type systems really need a full semantic tree to do anything meaningful due to the complexities of compile time code and type instantiation.

I wouldn’t say this depends on abstract complexity of “type system”. For example, Rust is pretty advanced, but lazy query architecture works well for it. Usually, it’s the name resolution/macro expansion that puts the wrench in the works, not a Turing-complete type system.

That being said, yes, in Zig you probably can’t do lazy query-based IDE. It really is a compile-time smalltalk, and wants to have an image.

I guess I meant complexity in the sense of symbol resolution in this context. So things like scoped type aliases, expressions evaluated as template or trait arguments, that sort of thing.

This isn't the case for Zig, because the compiler works off a graph of declarations. You only need to compute the declarations referenced in a code block (which have well-defined sources, since you cannot create new decls at comptime right now) in order to compute the function itself.

>What if the completion source is in a different file altogether?

Well, yes, this is a very common case. I don't believe matklad is arguing that the computation should only be restricted to one file. From what I can gather from the article, these "working" and "ready" copies would be per-file as well and the "ready" states of each file would be invalidated as a referenced file gets edited.

I doubt you need a full semantic tree. Just the parts that are accessible to the local edit scope. Like if I break the code within a function, all other code in the unit should be ok. It gets trickier with broken syntax but there’s a lot of clever recovery techniques I’ve seen.

How do you propose code completions are done without it? To even produce a set of suggestions, you need to understand the parse context, the set of available symbols with appropriate types and scope resolution. AST parsers already have recovery mechanisms, and I am assuming those are all working as intended.

When I’m in function foo I may have a local closure X. When I’m editing bar I don’t need to consider X, right?

I think what you’re trying to say is “how do you pick a subset of a tree to load at any given point” / “it’s simpler to have the entire semantic tree and traverse it than trying to keep a live tree”. If that’s a correct reading, I agree, it’s a difficult problem. I don’t have any specific recommendations other than to note it’s a graph and graph databases don’t need to keep the entire graph in memory to do queries so there must be something similar you could do with code. The other part is that not all parts of the graph are equally relevant so intelligently pruning it should result in better completions. I explored this mildly as an undergrad but the topic never sufficiently interested me to continue pursuing it.

If foo is a function in this example, then yes I agree for most PLs. If foo is a class, namespace, or some other declaration context however, its contents are definitely relevant when editing a different context due to possible usage of scope operators.

Reply on news.ycombinator.com