diffxx

Completely agree. That is why having built a tool similar to this one, I'm not even linking to it. The complexity involved in working around the OS limitations is maddening and convinced me that it would be better to think of a different approach to writing software that wouldn't require monitoring files to achieve the fast feedback loop that these tools are designed to facilitate.

The magic file approach described by kevincox below is probably the best way to get > 95% of the benefit with < 1% of the work.

What do you mean by beat kqueue? Is it faster than kqueue? Does it use less memory than kqueue?

How does the baseline filesystem watcher work? If it doesn't use kqueue, does it poll the filesystem periodically and diff against an in memory representation? If yes, see my other comments. If not, I am genuinely curious what you are doing because you know something that I do not.

What do you mean near-zero? You said that inotify doesn't work (and ned14 offers his comments about it). If you are using polling, I do not understand how your approach could be using non-zero amount of resources. Let's say you are monitoring a directory with 1 million files, how can you store the state in less than 20MB of memory (which is about the most optimistic lower bound that I can think of)? What is your secret sauce? Do you mean there is no overhead beyond the baseline watcher? But what about the overhead of the baseline watcher itself?

For what it's worth, in spite of ned14's comments, I have never seen inotify fail in practice (except for if it hits the os file descriptor limits in which case it does fail noisily). The tool I wrote uses inotify for linux. It is used by thousands of developers every day as part of an editor integration and there are no open issues about dropped file events.

Your time frame is probably about right. It took me about a year to work through all the edge cases.

I have personally written a similar tool and I am very curious about how this could be using a near-zero amount of resources while maintaining accuracy. As far as I know, there are two ways to implement this functionality: 1) store an in memory representation of the file system and periodically refresh the in memory state by polling the paths under watch and emitting events when differences are detected 2) hook into the underlying kernel events like kqueue, inotify, fsevents, ReadDirectoryChangesW, etc and report events

Option 1 uses a lot of CPU and memory (the map storing the paths being monitored could easily grow to be tens or even hundreds of megabytes if many files are being monitored, which is often the case in large source projects). I have seen tools that use polling with a 100ms interval continuously burn 50% of cpu monitoring a modest sized directory with tens of thousands of files.

Option 2 theoretically would use less memory and little to no cpu, but in practice, the story is more complicated. If you are using an inotify or kqueue like api, you will have to store handles for all of the paths that are being monitored, which can take a significant amount of memory. On macos, the file system events are not accurate in the sense that you can't trust the type of event. It doesn't reliably distinguish between creation and modification events. So if you want to know specifically what kind of event happened, you end up back in case 1 where you have to store an in memory representation of the file system and diff against the in memory representation and the current file system state when you detect an event. For some use cases, you may not care to distinguish between creations and modifications and can get away with a lower memory, but less accurate, solution.

In my experience, getting all of this right is much more difficult than it appears at first glance. Good luck to you.

Yikes, this hit very close to home for me.

Yeah, this is difficult, though not impossible, to get right with the inotify/kqueue api. Windows and mac do provide apis for recursive directory monitoring but a cross platform tool will have to solve this problem. At a high level, the way to do it is to create an in memory representation of the file system that caches a watch handle for every file. When a deletion of a file is detected, you must create a watch on the parent directory, if there wasn't one already. Then you should be able to detect the ensuing create. To make this more concrete, the problem that a file watcher needs to solve is the problem of keeping its in memory representation of the file system consistent with the actual file system. Watch events are just a useful side effect of this process.

The other fun part is that there is often a lag between the deletion and the create in the text editor case so it is necessary to defer triggering events when a deletion is detected and wait a little while to see if there is a corresponding creation. Otherwise you may rerun your command that depends on the file before it exists.

It is possible to get like a 99+% solution to this problem without polling but it is a lot harder than what these simple tools, including entr do. The upshot is that file monitoring should be looked at as a lowest common denominator solution. A better solution is to build automatic command running into the text editor itself.

Regardless of intention, the inability of the author to adequately explain why a fork was necessary, apart from some hand wavy arguments that some other solutions were inadequate, does not inspire confidence. I find it very hard to be generous with the author given the way that he paints himself as a visionary while casually dismissing the work and perspective of the people who actually did the real visionary work of seeding the technologies that he has hitched his wagon to.

This comment perfectly encapsulates the point that I am making about best practices: the concept is used as a cudgel to silence debate and to confer a sense of superiority on the practitioner of "best practice." It is almost always an appeal to authority.

No one wants cowboy pilots ignoring ground control. Doctors though do not exactly have the best historical track record.

Knowledge communities should indeed work towards consensus and constantly be trying to improve themselves. Consensus though is not always desirable. Often consensus goes in very, very dark directions. Even if there is some universal best practice for some particular problem, my belief is that codifying certain things as "best practice" and policing the use of alternative strategies is more likely to get in the way of actually getting closer to that platonic ideal.

Very good points. The proprietary test suite is clearly the (open) secret to SQLite's success. It seems to me that it isn't even entirely accurate to describe SQLite as written in C when the vast majority of its code is probably written in TCL that none of us have seen. It's more like C is just how they represent the virtual machine which is described and specified by its tests. The virtual machine exists outside of any particular programming language but C is the most convenient implementation language to meet their cross platform distribution goals.

If someone did want to carve into SQLite's embedded db monopoly, it would take years to develop a comparable test suite. This seems possible, particularly if they develop a more expressive language for expressing the solutions to the types of problems that we use SQLite for. Who would fund this work though when SQLite works as well as it does?

> it's not that "best practices" or any of those things are what causes trouble; it's failing to recognize that they're just tools, and people will still be the ones doing the work. And people should never be treated as merely tools.

For me, the concept of best practices is pernicious because it is a delegation of authority to external consensus which inevitably will lead to people being treated as tools as they are forced to contort to said best practices. The moment something becomes best practice, it becomes dogma.

Fair points, although the bus factor is more like 3 or 4 for SQLite as far as I know. The question though is that if the entire team vanished from the face of the earth, what would the impact be? My guess is that either SQLite would be good enough as is for 99% of use cases and it wouldn't need much development apart from maybe some minor platform specific use cases or, if new functionality truly is needed, then it would be better for a new team to rewrite a similar program from scratch using SQLite as more of a POC than as a blueprint.

Also, Fossil is essentially implemented _in_ SQLite. Fossil is used to develop SQLite which is used to implement Fossil. It's a virtuous cycle. For the SQLite project, using Fossil is obviously superior to git. This doesn't mean that arbitrary projects should use Fossil over git.

> The few core developers they have do not work with modern tools like git and collaboration tools like Github, and don’t accept contributions, although they may or may not accept your suggestion for a new feature request.

The funny thing about this comment is that SQLite is as close to the gold standard of software quality that we have in the open source world. SQLite is the only program that I've ever used that reliably gets better with every release and never regresses. Could it be that this is precisely because they don't use "modern tools" and accept outside contributions?

That's not what I took away from the book. He mentions his own personal enjoyment of trashy television multiple times. His concern was that he believed that democracy is fundamentally incompatible with a media landscape dominated by television (s/televison/social media in 2022).

Lua is misunderstood because it looks like superficially similar to a number of other languages but has a unique value proposition that is rarely well explained. The frustration that ensues from trying to use lua as though it is javascript or python or c is real. Because the value proposition isn't well explained, one has to use it and gain experience with it to appreciate it.

One of the key differentiators for lua is that it has multiple implementations that are both rock solid and will work on essentially any computer that you are likely to ever work with. Luajit is api compatible lua 5.1 (with a few 5.2 features backported). Lua 5.1 came out in 2006, which is to say that when writing lua 5.1 code, you are targeting a language that hasn't really changed in 15 years. This is incredibly liberating because you can trust that any code that you write in luajit or lua 5.1 is likely to work essentially forever (modulo security issues of course).

Lua is also one of the most simple languages out there. Pretty much everything that you need to know is in the reference documentation, which is a single file that you can easily download and store for offline use. It also hasn't changed in a decade and there is no reason to expect it will in the future.

Simplicity begets simplicity. As simple as lua is, you can make it even simpler by restricting yourself to a limited subset of the language. The more you restrict yourself, paradoxically the more liberated you are. You eventually realize that you can express any computational problem in lua. Lua provides an implicit vm that serves as an idealized model of a computer. It has numbers, strings, arrays and associative arrays (with a uniform syntax for both types of arrays). This is enough to do pretty much anything without too much pain. The lua runtime itself is fast enough to do most programming tasks faster than you can perceive. For the remaining tasks, you can just implement them in c and call them from lua. But you won't be writing normal c, you will be writing your own dialect of c that itself reflects lua. Many of the problems of c, particularly around memory management, are avoidable when the c code is explicitly designed to be embedded inside of a lua program.

Continuing on the theme of simplicity, lua is also one of the fastest scripting languages with respect to startup time (EDIT apologies for the poor formatting below but console output doesn't work great with HN):

bash-5.1$ time lua -e 'print "Hello, world!"' Hello, world!

real 0m0.004s user 0m0.002s sys 0m0.002s

bash-5.1$ time python3 -c 'print("Hello, world!")' Hello, world!

real 0m0.034s user 0m0.024s sys 0m0.007s

bash-5.1$ time node -e 'console.log("Hello, world!")' Hello, world!

real 0m0.062s user 0m0.049s sys 0m0.011s

bash-5.1$ time ruby -e 'puts "Hello, world!"' Hello, world!

real 0m0.086s user 0m0.062s sys 0m0.021s

The lua startup time is actually comparable to a standalone c executable:

bash-5.1$ time printf '#include <stdio.h>\nint main() { puts("Hello, world!"); return 0; }' | gcc -x c -o a.out - && ./a.out "on mac os, there are some checks that are run the first time a new exectuable is called so do one dry run before timing the results" > /dev/null; time ./a.out

real 0m0.074s user 0m0.050s sys 0m0.021s

Hello, world!

real 0m0.002s user 0m0.001s sys 0m0.001s

Comparing startup time may seem silly, but it raises the question how exactly you are benefiting from the excess complexity relative to lua that those other language runtimes have. You also can exploit the fast startup time to create a low latency development experience with minimal tooling. Configure your editor to save lua files with every keystroke. Then in a shell window, run something like: > ls foo.lua | entr -c time lua foo.lua

You might be amazed by how much you can do in a lua script before the the latency becomes noticeable. By timing the results, you are also immediately aware of the impact on performance that any change to the script might have. This enables you to have an entirely different kind of development experience in which you are always fully aware of the state of your program because you are never waiting for compilation. The program either does what you expect it to, or it doesn't. No time is spent in the liminal state where one suspects the code may be correct (or incorrect), but you aren't really sure. You start realizing that if you organize your code in a certain way, you almost never have to wait more than say 100ms to get feedback.

On top of all that, if you don't like the syntax of lua, you can write a parser for another language in pure lua. Your new language, which will transpile to lua, will automatically inherit the good properties of the lua runtime.

Mastering lua will completely change the way that you think as a programmer and has the potentially to radically improve your experience of programming. Of course, once you have mastered lua the next step is to go beyond lua...

Define human.

The paradox of copilot is that anyone who is skilled enough that they can be responsibly trusted to use it doesn't need it.

> I agree that Rust is a large language, but I think you're overstating the complexity: it's been my experience that you don't need to know all (or even most) of the "clever" stuff to write large, performant Rust programs.

The same could equally well be said for, say, scala. In my experience though, you always pay one way or another for the unneeded and unwanted features in a language. Unreasonable compile times that kill joy and productivity are just one of the ways.

Gavin, there is no point in arguing with fanatics. (You have to know that none of them are actually going to dive into your code and possibly learn something new that didn't come from a vetted authority figure). They are incapable of listening to their own independent thoughts, assuming that they have them, and must attack anyone who counters their dogma because it risks shattering their core beliefs, which is intolerable. The funny thing is that the truth of rust is right there in its name. It is a corrosive process that ruins the iron that it touches.

For anyone interested in the arguments in this piece, Deschooling Society by Ivan Illich is highly recommended.

As impressive as the tree-sitter ecosystem is, there is a certain danger in building tooling on top of it. Unless tree-sitter is used as the parser in the actual language compiler, it is almost always going to be an approximation to the language grammar. This is somewhat risky if it is important that any kind of semantic analysis, including diffing, is correct. In the case of difftastic, what worries me is that it might be possible to inject malicious code into a repo by exploiting a bug in the tree-sitter grammar/parsing implementation such that the malicious code is hidden from difftastic because, for example, it is treated as whitespace when it actually isn't. This of course wouldn't be a problem if PRs are being handled by github, but what if down the road github starts offering difftastic (or something similar) as a diff option for PRs?

Admittedly, this is a somewhat far fetched scenario that I have posited but I am somewhat concerned that the proliferation of incorrect tree-sitter grammars is going to lead to the worst kind of problem down the road: tools that work great _most_ of the time.

> I don't understand what the issue is with cross-compiling software, especially for different architectures.

Agreed. The cosmopolitan project is incredible hacking and I appreciate the author's ingenuity, but this has always felt a little bit to me like a solution in search of a problem. How is cosmopolitan software intended to be distributed? If the answer is on the internet, I don't see how a single fat binary helps the end user dramatically more than separate download links. For the developer, cross compilation can be automated as part of deployment. The platform specific binaries should both be smaller than the universal ape binaries and won't resort to hackery to work properly.

> people are just inherently greedy. While other political systems attempt to shut our nature down capitalism embraces ourselves and aligns our incentives to generally make society better.

I understand where you are coming from but I will state unequivocally: people are not _just_ inherently greedy. Each of us has the capacity to be greedy and selfish, but we also each have the capacity to be giving, caring and accountable to others. How might we act if we lived in a society in which wealth hoarding was a cultural taboo rather than a celebrated cultural value? What if we looked towards billionaires with pity rather than envy or resentment?

As the late anthropologist, David Graeber, put it: “The ultimate hidden truth of the world is that it is just something we make, and could just as easily make something different.” Rather than just concede that capitalism is the best we can possibly do, why not consider how we could reform it and/or replace it with something better? (For the record, I consider communism and capitalism two sides of the same coin. We need to be more creative and visionary about the world we want to live in than what these tired ideologies offer.)

Actually the original comment makes complete sense. To only give tickets for drunk driving essentially makes it so that drivers have blanket immunity from any criminal liability for their behavior, no matter how reckless, so long as they are not drunk on alcohol. By rarely holding reckless drivers responsible for their behavior, we are essentially incentivizing that behavior.

From the car's perspective, the current law is great. Many people who would consider alternative transportation such as cycling or walking are discouraged by the fact that these activities are much more dangerous for you than driving. If you get hit by a reckless SUV at 40mph while you are in a car, you have a pretty good chance of surviving. Your chances of surviving as a pedestrian or cyclist are drastically lower. Rationally, many people then opt for the car (for me, must American roads are too dangerous for cycling so I only walk to get around even though I do enjoy cycling). This is a vicious cycle that ensures that cars will perpetually dominate our streets. I fear self driving cars (or an infinite series of failed attempts at self driving cars) are only going to make this worse. For all of the benefits that cars do offer us in our daily lives, from my perspective, it seems clear that the scales have tipped way too far in favor of the car.

Different legal regimes are possible. For example, we could change the law so that by default drivers are assumed responsible for any collision they were a part of. In the case of a driver killing a pedestrian, the burden should fall on the driver to prove that it was the pedestrians fault. The current enforcement of the law seems to almost imply that pedestrians and cyclists are suicidal so if they die in a collision, it must have been their fault somehow. Surely we could require drivers to drive in such a way that they have very low probability of injuring anyone? Or would that be an unimaginable infringement on the rights of the driver (and car)?

The beauty of a system that actually holds drivers accountable for the consequences of their actions is that we actually wouldn't have much need to enforce laws like speeding and running red lights because few would want to take the risk of losing their license if they were involved in a crash.