maxbrunsfeld

This is an amazing summary. Thank you.

It generates LR(1) parsers, and can use GLR on an opt-in basis, for handling specific conflicts.

One more that I know of:

Atom understands your code better than ever before - https://news.ycombinator.com/item?id=18349013 - Oct 2018

Not that I know of, right now :(.

In the near future, we'll create some more GitHub-specific documentation that walks you through how to add advanced language support for any programming language on GitHub, by writing a Tree-sitter grammar, and then by writing the tree queries that are used for syntax highlighting, simple code navigation, and someday soon... precise code navigation.

At GitHub, we're in the process of building a more precise code navigation system on top of Tree-sitter, that models language-specific name-resolution rules in detail.

Our currently-available code navigation system also uses Tree-sitter, but it is pretty simple; it just matches up references and definitions by their name.

Most tree-sitter grammars just parse comments as a single token. Can you give an example of what you mean when you say "contents of the comment parsed out"?

Are you talking about conventions like JSDoc, for putting structured data inside of comments? On GitHub, we handle that by parsing JSDoc comments in a separate pass, using a separate parser. We do it this way because JSDoc isn't really part of the JavaScript language, not all projects use JSDoc, and not all applications are interested in parsing the text inside of comments.

I think that it'd be pretty easy to generate parser code in other languages besides C, but it would be a lot of work to do to port the core library itself[1] to those other languages.

[1] https://github.com/tree-sitter/tree-sitter/tree/master/lib/s...

I agree about the Objective-C grammar! Although it looks like somebody's started work on it:

https://github.com/merico-dev/tree-sitter-objc

That's great to hear. Thanks!

One day, I would love to generalize the web-based playground so that you could edit the grammars. But it's complicated, because we use C as our output language, so you would always need to recompile the C after changing the grammar.

So, I would say that it's not on our near-term roadmap.

Hey, Tree-sitter author here. Thanks for posting! Let me know if you have questions about the project.

I think there's a lot of open-source projects that are somewhere in between the two extremes presented in the article (communities vs makers-and-fans). As a maintainer of a project with more moderate user base, it's a mix. I have a lot of interactions with contributors that are very rewarding to me, and also add genuine technical value. On the other hand, there are also a lot of contributions that are more of like "support requests", where I feel like I need to respond to them because it's my responsibility.

I would guess that the bigger a project's user base gets, the greater the proportion of interactions that are more like makers-and-fans.

I like this solution, and have thought about adding something similar as an "advanced feature" in Tree-sitter grammars. I'm interested to see how you've used it in grammars.

Yeah, I agree. To really demonstrate the best possible IDE error recovery that LR/GLR can provide, pairing a robust batch error recovery strategy with Wagner's (and your and Lukas's) history-sensitive approach is probably the way to go.

At the time that I was implementing error recovery for Tree-sitter, that felt beyond my "complexity budget", since batch error recovery was already not trivial, and the batch approach was good enough for the features I was building on GitHub/Atom. Someday, I could imagine augmenting it with history-sensitive behavior. I am thankful that Lukas has helped to "pave that path" so thoroughly.

While Tree-sitter is heavily influenced by Wagner's thesis, our error recovery strategy actually uses a novel approach, so it is fair to question whether Wagner's would have different properties.

For error recovery, Tree-sitter is currently optimized for some goals that are a bit different from the goals of a single-language IDE like Rust-Analyzer:

  * Like RA, we want to minimize the impact of syntax errors on basic features like syntax highlighting and symbol name extraction (which are important to GitHub, and in lightweight text editors like Neovim and Atom).
  * Unlike RA, I wanted to avoid requiring grammar authors to think about error recovery at all. 
  * We needed to tightly control the performance cost of pathological error recovery cases, like parsing a large Ruby file as Go (either accidentally or adversarially).

My understanding is that Wagner's thesis is mostly concerned with "history-sensitive" error recovery in an IDE: using earlier syntax trees as an additional input to the error recovery process in order to better understand the user's intent when editing.

I think this idea makes a lot of sense, but it doesn't address cases like Matklad has prevented here, where the IDE is presented with already-invalid file.

> DeepSpeech v0.6 with TensorFlow Lite runs faster than real time on a single core of a Raspberry Pi 4

This is great news.

I'm not very familiar with the deep learning framework ecosystem. Does anyone know what the simplest way would be to incorporate a DeepSpeech model into a WebAssembly project?

For instance, is it straightforward to compile Tensorflow Lite with emscripten? If not, can this TensorFlow model be run with tensorflow.js? Or can it be converted to some other format that is easier to use with WASM?

Many parser generation tools use their own custom grammar language, and then generate a C parser based on that. With Tree-sitter, it’s a similar setup, except the grammars are written in JavaScript instead of some custom language.

The parser generator itself is all written in Rust, but the end user doesn’t need to use rust in any way.

The baseline memory footprint of Electron is already quite reasonable. See https://news.ycombinator.com/item?id=21347575

Do you have some packages installed? If so, can you re-run with `atom --safe`?

Granted, it is totally normal to have packages when using Atom, but their memory cost shouldn't be considered a penalty that you pay due to Electron. Rather, it's a penalty that you pay if you want an extensible application in which extensions are allowed to use JavaScript.

It is not because of Electron.

An empty Atom window containing 5 source code files consumes about 120MB of RAM. For comparison with native apps, an empty iTerm2 window uses 100MB, and so does an empty Notes.app window. The base memory footprint imposed by Chromium is not that significant.

Yes! I'm very interested in making the Tree-sitter runtime and all of the parsers available as pre-built WASM modules. I think it should be straightforward to do.

I am not super familiar with all of the variations on LL(k) parsing, but my understanding is that the reliance on multiple tokens of lookahead makes incremental parsing a bit more complex, whereas it fits very cleanly into the framework of LR(1)/GLR parsing.

My hope is to make parser development with Tree-sitter easy enough that you wouldn't have to pave a new path for your structured editor (unless you wanted to) - you could just create a Tree-sitter grammar for your DSL and implement your custom editing logic as an Atom package.

I have tried to make Tree-sitter more approachable than, say, Bison in a several ways. Usually, once a language's basic structure is in place, the process of adding a new feature is pretty declarative and easy. But it's still true that in the course of developing a grammar from scratch, some understanding of LR is important.

Tree-sitter isn't really an alternative to LSP. We think of it as solving a different set of problems.

LSP is probably the best way to provide the fixed set of classic IDE features that LSP does - inline diagnostics, autocomplete, go-to definition, etc.

We think that for many other features, Tree-sitter is a much cleaner solution than pushing the logic out into the language server. For example, if you want accurate syntax highlighting that's lightweight and updates immediately (as opposed to in a delayed fashion like in most IDEs), you need incremental parsing. Incremental parsing is a highly specialized problem, so writing an incremental parser in the form of a Tree-sitter grammar is much simpler than modifying a language's compiler toolchain in order to make it incremental, and bundling that whole toolchain into an app like Atom.

The same goes for other features besides syntax highlighting. The syntax tree is now available via a uniform, in-process API in Atom, and is always up-to-date, so you can script the editor to manipulate your code intelligently. I'm not totally sure what kinds of things we'll end up building on top of that, but I think it's a different set of things than LSP will end up providing.

Hello! I worked on this feature and wrote the post. I'd love to hear any feedback or answer any questions.

> GPUArrays never had to implement automatic differentiation explicitly to support the backward pass of the neuronal network efficiently. This is because Julia's automatic differentiation libraries work for arbitrary functions and emit code that can run efficiently on the GPU. This helps a lot to get Flux working on the GPU with minimal developer effort - and makes Flux GPU support work efficiently even for user defined functions. That this works out of the box without coordination between GPUArrays + Flux is a pretty unique property of Julia

Every time I read about Julia, I’m amazed. What a game changing tool.

The Externals API is designed to only let you do things that are compatible with incremental parsing. It’s good for things like indentation based delimiters and automatic semicolon insertion.

You cannot call back into the parser; it’s just for tokenization.

You can produce zero length tokens, and we do use this feature a lot.

You dont have access to the parse state directly, but the external scanner is passed an array of Boolean values that indicates which external tokens are expected in the current state.

Thanks for the kind words!

I haven't specifically pursued integration with LSP, but Tree-sitter has been used to build a couple of language servers which work with both VSCode and Atom (and probably other editors):

* Bash - https://github.com/mads-hartmann/bash-language-server * Ruby - https://github.com/rubyide/vscode-ruby/tree/master/server

Yeah, the grammar author does fully control the division between the parser and lexer: every literal (string or regex) in the grammar corresponds to a token. There's also a `token()` function that you can use to specify that an arbitrary rule should be handled by the lexer as a single token.

In most cases, you don't have to think about it; the obvious way to write something is the right way. There are cases where its helpful to have a mental model of how lexing works.

Good question.

At one time, I had the thought that folks who disliked JavaScript might want to write their own grammar DSL in another language, which could generate compatible JSON to be consumed by the core Tree-sitter compiler library. Now, I think this flexibility is pretty unnecessary.

I've also thought that some applications might want to inspect the grammar JSON at runtime in order to do some kind of meta-programming. I haven't really thought that through though.

In either case, versioning would probably be a good idea. The actual parser is already versioned so that we can error out if you try to use a parser with an incompatible version of the runtime. We could just add that same version number into the grammar.

It's not actually implemented as a scannerless parser, but the grammar API mostly abstracts away the parser/lexer distinction.

We have to handle JSX and all language versions combined because for our use case, users need to be able to open up `.js` files and have them Just Work.

It's a similar story with Python and Ruby - we have to handle the union of all language versions.

Hey, author of Tree-sitter here. Thanks for sharing this! I'd be happy to answer any questions people have about the project.