587 karma · joined August 16, 2022
Of course it's much easier to just globally turn on release mode and optimize everything whether it matters or not. This both slows down the development process considerably and leads to bizarre issues like this.
I'm not saying that incremental is bad per se, but that the choice of guaranteeing incrementalism complicates things for cases where it isn't necessary. I am not super familiar with lsp, but I can imagine lsp having a syntax highlighting endpoint that has both batch and incremental modes. A naive implementation could just run the batch mode when given an incremental request and later add incremental support as necessary. In other words, I think it would be best if there were another layer of indirection between the editor and the parser (whether that is tree-sitter or another implementation).
Right now though, you have to opt in whole hog to the tree-sitter approach. As mentioned above, incrementalism has no benefit and only cost for a batch tool like difftastic or semgrep to mention two named in this thread.
I was too vague on unicode. I meant unicode character class matching. Last I checked, tree-sitter seemed to use a binary search on the code point to validate whether or not a character was in a unicode character class like \p{Ll} rather than a table lookup. This lead to a noticeable constant slowdown of like 3x at runtime if I recall correctly for the grammar I was working on compared to an ascii set like [a-z].
At the same time, I believe that there needs to be a corrective about what tree-sitter should and should not be used for. There are companies building security products on top of tree-sitter which I think is an objectively bad idea given its problems and limitations. Difftastic is to me a grey area because it could lead hypothetically to a security issue if it generates an incorrect diff due to an incorrect tree-sitter grammar. Unlikely but not impossible.
Your point about batch vs incremental is spot on, though even for IDEs, I think incremental is usually overkill (I have written a recursive descent parser for a language in c that can do 3million lines per second on a decent laptop which is about 60k lines per 20 ms, which is the window I look to for reactivity). How many non-generated source files exceed say 100k lines? Incremental parsing feels like taking on a lot of complexity for rather limited benefit except in fairly niche use cases (granting that one person's niche is another's common case).
That being said, it is impressive that their incremental algorithm works as well as it does but the cost is that grammar writers are forced to mold a language grammar that might not fit into the GLR algorithm. When it doesn't work as expected, which is not uncommon in my experience, the error messages are inscrutable and debugging either the generator or the generated code is nigh impossible.
Most of the happy users have no idea how the sausage is made, they just see the prettier syntax highlighting that works with multiple tools. I get that my criticism is as welcome as a wet blanket, but I just think there is something much better possible which your comment hints at.
For syntax highlighting and code folding, what I said above is probably generally fine. Except for the fact that it will probably be wrong from time to time and then downstream users will waste hours of their lives trying to fix the broken highlighting instead of actually working on their real work.
On the implementation side, the whole thing has bizarre choices. It has created a culture in which checking in enormous generates files is common. It abdicates common and important features in popular languages like significant whitespace to error-prone external scanners. The GLR algorithm sometimes fails inexplicably and is essentially impossible to debug (not sure if the bugs are in the algorithm or the implementation). It bizarrely uses rust and node. The js dsl for grammars is needlessly verbose. The codegen is fairly slow in spite of using rust and the generated c files are insanely huge (probably a consequence of the GLR algorithm) which leads to horrific compile times which makes incremental development a nightmare. It also uses (unless it's been updated in the last six months) a terrible algorithm for unicode lexing.
In my opinion, the whole premise of tree-sitter is kind of wrong. What would be better would be a uniform dsl for generating an AST (which can easily be represented with c structs or classes in any oo language). Trying to make a universal glr parser for all languages is a bad idea. It would generally be much easier, more accurate and faster to write the AST in the dsl that external tools recognize and then handwrite the parser that generates the AST. In the best case, the compiler already exposes an api to get the ast for some source code and you just need to write a translation program from compiler format to dsl format. Worst case, you rip out the parser in the compiler and have it generate the dsl ast format. Most handwritten parsers are only a few thousand lines of codes (maybe 10k at the high end).
Tree-sitter is great until you inevitably run into a problem with it and find that it is impossible to fix properly if you can even figure out why the problem even exists.
The idea of a uniform api for querying syntax trees is a good one and tree-sitter deserves credit for popularizing it. It's unfortunately not a great implementation of the idea.
To go even more meta, there is an analogy I'm trying to make right now in which I am visualizing a road and thinking about how describing the road relates to the process of writing. In my mind's eye, I can see the full length of the road and all of its contours but I can't actually describe the individual stretches of the road coherently without enumerating them. Something similar happens with writing. I can visualize what I want to say far beyond the next word, but it's true that the actual process of writing goes word to word, much like how the process of token selection is described for an LLM. The question is whether the LLM has an analogous conception of where it is going. Going back to the process above, sometimes I know where I am going and haven't yet figured out how to articulate it yet. It is through the process of writing that I am able to articulate that thought. But the thought preceded my articulation of it. I don't know to what extent LLMs have coherent thoughts that they are articulating or if that even makes sense for the type of intelligence they project. My suspicion is that they don't have additional sensory inputs beyond language that give thoughts the immaterial shape that then is expressed in language. Without that, I am skeptical that they will truly get beyond regurgitating and/or remixing what has already been fed to them textually. That doesn't diminish how amazing they are, but I am somewhat more in the Brooks/Knuth camp that they are impressive and surprising, but there is something that ultimately leaves me a bit cold about them.
Stories of deep psychological truth that transcend physical space and time. Just like all great religious stories.
Even if this is true, should this be celebrated even it comes at the expense of the > 99% of the population that doesn't work for Lyft as well as the thousands of drivers who are being exploited by their business practices? It is a relatively small minority of citizens who actually rely on these businesses for their day to day lives but their externalities are borne by all.
> The idea that taxi dispatchers could be more efficient, or as efficient, is ludicrous. Does nobody remember regularly waiting 30-60 minutes for a taxi from a dispatcher?
Even if you're right, this presupposes that it is a net benefit to have easily available taxis. It also seems to suggest that efficiency is best measured by latency between ordering a taxi and being picked up. Taxis are easily the least efficient form of transit. They are even worse than private vehicles because of the deadheading. Then there is the whole induced demand problem where they actually make traffic worse as people with the means opt out of more efficient mass transit so overall transit efficiency suffers even if it is (seemingly) more convenient for many users.
> In any case, yes surely Lyft could cut costs drastically in order to become more profitable, but it's also continuing to invest in growth as much as possible.
So they can try to hold the public hostage and mobilize their minority user base against any legislation that will rein in their excesses.
Call me a pessimist, but I don't think it's possible to eliminate murder so I think that is an unreachably high bar.
That being said, I think that there is lots of low hanging fruit that we absolutely could do something about. Traffic fatalities and injuries are an obvious one. In the US, if we collectively decided that it is unacceptable to have tens of thousands of people die every year in auto incidents and scores more seriously injured, we could rework our road infrastructure and impose regulations that would drive these numbers down dramatically, even if not to zero.
The biggest challenge to enacting the types of reforms that would improve most people's quality of life is that advocates have mostly done a poor job of communicating their positive vision for how such changes would improve most people's lives. The story progressive transit advocates (myself included) tend to tell sounds like it's about taking away the rights and freedoms of drivers. This is undeniably true. Freedoms and rights are always in tension. Progressives will always lose though if they aren't able to tell a compelling story about the benefits of the changes they wish to impose. Otherwise the other side will be able to successfully hammer them on whatever is being taken away.
In our current regime in the US, we have essentially legalized murder if you are driving a car. If someone runs me over with their SUV while I am legally crossing the street because "they didn't see me," they are extremely unlikely to face any consequences unless they were drunk. But at the same time, if I'm driving a car a lot, I also face the risk of killing or harming someone else even if I am always doing my best. This would be devastating even if I wasn't legally at fault. In other words, the freedom to kill someone with no accountability is no freedom at all.
The point is not that I want people to go to jail for killing people with their cars. I want to live in an environment where people are not killed by cars. There are many things like this where I think there is a more positive vision for society that is not that hard to imagine but that does require letting go of some of our past ways of looking at things. In this particular case, a century of marketing and pr by auto/oil companies has warped our perspective that we can't even see how anti-human the industry and our infrastructure has become. That doesn't mean we have to get rid of cars entirely, but we should rethink our relationship with them. There are many things like this.