Hacker News Reader

Comment by inshadows | Hacker News Reader

Entr and many other similar alternatives do not satisfy my needs. The main fault is that they take list of files instead of root directory and possibly include/exclude patterns. So when I create or rename files, these tools either don't register new files, or they just fail on missing files.

I'm rather reusing existing stuff but seeing how every scriptable filesystem watcher misses the point, I'm inclined to write my own inotifywait/inotifywatch wrapper.

View on HN

That's literally in the article; `entr -d`. The manpage also has an entry. http://eradman.com/entrproject/entr.1.html#d

Terminal 1:

    mkdir /tmp/project
    cd /tmp/project
    touch file{1,2}
    ls | entr -d echo "a change"

Terminal 2:

    rm file2

Terminal 1:

    entr: cannot open 'file2': No child processes

Regarding `-d`:

> Track the directories of regular files provided as input and exit if a new file is added. This option also enables directories to be specified explicitly. Files with names beginning with ‘.’ are ignored.

So first, it doesn't track NEW directories, and second, it exits if new file is added. Exactly how is this useful?

EDIT: All I want from filesystem watcher is to track files by pattern and just re-run the command, however complex that may be to implement via OS interfaces. When I'm working on a Python project that has packages (directories), I may also refactor (rename files), and this simplistic behavior does not catch that.

> All I want from filesystem watcher is to track files by pattern and just re-run the command, however complex that may be to implement via OS interfaces.

I've had good luck with modd: https://github.com/cortesi/modd/

I just tested exactly what you described, did not see any errors, and it ran my commands exactly the way I wanted.

I can attest to modd too, it uses a (per project) configuration file to define include/exclude (recursive) globs, along with respective blocks of commands.

I would also like to extol virtues of its sister project devd[0] from same author which makes web development palatable to me. It's a little websocker server that injects a tiny script into HTML to reload the page in browser once modd detects file change, rebuilds front-end and sends devd a SIGHUP.

[0] https://github.com/cortesi/devd

I think the interface for `-d` is clunky, but here is the way it is intended to be used:

    $ while true; do
    > ls -d src/*.py | entr -d ./setup.py
    > done

that way when a file/dir is added/removed it restarts

This is in fact the documented recommendation, from the manpage:

  Rebuild project if a source file is modified or added to the src/ directory:
    $ while true; do ls src/*.rb | entr -d make; done

> All I want from filesystem watcher is to track files by pattern and just re-run the command, however complex that may be to implement via OS interfaces.

I assume I'm misunderstanding something, so this is probably a naive question and maybe you can explain further, but if that is all you need why not just write it yourself? It sounds like an <1 hour project to write something cross-platform in Qt that monitors the filesystem recursively with QFileSystemWatcher https://doc.qt.io/qt-5/qfilesystemwatcher.html checks for the configured regexen and runs the configured command.

What's wrong with watchman? Seems to be made for that very use case.

https://facebook.github.io/watchman/

I have implemented stuff like this. It is not even remotely a <1 hour project. Not if you want it to work reasonably well and not be broken all the time. It’s a full workday for someone who already understands the problem. There are a number of cases you often want to handle:

- Multiple files may change in quick succession, for example, if you hit “save all” in an editor. You might want to delay triggers to see if more events arrive.

- You will sometimes see incomplete / corrupted versions of files, because you ran a command while another program was writing the file. The other program probably should atomically rewrite the file, but you’re stuck fixing the problem. The user doesn’t want to see errors from this and you generally want to rerun the command.

- You need to execute commands in the correct order.

- A command may change state from being queued&ready to having outdated inputs.

If it’s just one command, sure, you could probably do it in an hour. But if there’s more than one command (usually the case!) then it gets far more complicated.

Just looking at the last time I implemented something like this--it was watchman + about 500 lines of code figuring out which actions to run and when. And that’s not even with any parallel execution.

I've ran into this issue myself and it's not filesystem watchers missing the point, it's actually something not supported in a number of operating systems. Thus you then either have to resort to nasty user-space hacks (which will be comparatively slow and resource hungry) or accept that you can't watch nested directories.

A more technical description is mentioned here: https://github.com/fsnotify/fsnotify/issues/18

inotifywait can do this just fine. Here's an emulation of `entr` with an `include pattern`:

    inotifywait -rq -m -e close_write --format %f . | grep '\.rules$'

Sure, there may races due to limitations of underlying OS interfaces, but no such races really matter when you edit with mammal speed, so it's perfectly sufficient, and so it is perfectly possible to make good file system watcher, without any user-space power-hungry hacks.

inotifywait is a user-space program that borrows it's name from inotify (because it uses inotify) and when using the -r (recursive) flag it sets up multiple inotify watches to work around the very problem I just described.

In fact it's man page comes with a big fat warning about using -r:

> Warning: If you use this option while watching the root directory of a large tree, it may take quite a while until all inotify watches are established, and events will not be received in this time. Also, since one inotify watch will be established per subdirectory, it is possible that the maximum amount of inotify watches per user will be reached. The default maximum is 8192; it can be increased by writing to /proc/sys/fs/inotify/max_user_watches.

(ref: https://linux.die.net/man/1/inotifywait)

Really what would be nice would be for Linux to support this natively so we don't have to resort to dirty hacks.

> inotifywait is a user-space program that borrows it's name from inotify (because it uses inotify) and when using the -r (recursive) flag it sets up multiple inotify watches to work around the very problem I just described.

What problem?

What you quoted is totally irrelevant for my workflow. On my crappy laptop and all projects in my work directory, the watches are established almost instantly, without any spike in CPU.

There are 4677 directories in Linux kernel and you can just put a knob change in /etc/sysctl.d/ if your code base is bigger than kernel.

You're just inventing OS limitations to justify limitations of lazy user-space programming.

EDIT: Setting up watches on kernel tree including directories in .git:

    ~/src/linux % inotifywait -r -m -e close_write --format %f . |& ts
    Jul 01 16:03:10 Setting up watches.  Beware: since -r was given, this may take a while!
    Jul 01 16:03:10 Watches established.

Less than 1 second. Big deal!

Your workflow might be ok, others might not. But blaming developers for being "lazy user-space [programmers]" when Linux lacks a feature that Windows and macOS support, and which could cause complications when worked around, is a really unproductive way to hold a discussion on HN.

You still didn't specify which feature Linux lacks that Windows and macOS supports. I have shown you that there's no need for any workaround, that the OS feature works as intended, but it seems you're keen on believing FUD scraped from web pages of lazy user-space programmers instead of considering my arguments and evidence.

FYI: What OS interface do you think that `entr` uses on Linux? That's right, inotify.

You're being quite aggressive about your disagreement here. I don't think it's "FUD" to rightfully point out that using inotify on a large directory tree takes extra userspace code that is explicitly called out in the inotifywatch docs as performing poorly enough to worry about race conditions. Not to mention that it requires a limits tweak. This isn't "lazy," it's wishing for the mechanism to be more capable. Would you have this same attitude if the default limit were 1024? 512?

Why wonder what my attitude would be with lower limit? Default limit is 8192 and it's OK for use case exemplified by entr and you can easily change it.

Not the original poster, but Windows allows you to get all changes in an NTFS volume, effectively allowing you to monitor notifications to an unlimited amount of files and directories (didn't remember what the feature was called, a quick googling led me to https://docs.microsoft.com/en-us/windows/win32/fileio/change... ).

This allows software like Voidtool's Everything to index the entire filesystem and update that index in real time. Using inotify for that is not possible since you'd have to recursively register for hunderds of thousands of directories, which far exceeds the limit of allowed registerations (and either way, would be quite wasteful to hold 100,000 registration handles). As a result, the Linux equivalent to Everything is unable to find recently created files, erronously finds deleted files and has outdated metadata fo recently changed files. Alternatively, you can attempt to reindex right before searching (e.g. rerun updatedb if you are using locate), but then your searches are much slower and the benefit of indexing for a quick search is reduced significantly.

A nicer API would allow to use one registration handle to get notifications for an entire directory tree or at least have a seprate API for an entire filesystem (like the NTFS option mentioned above).

At least, that was the conclusion I came to when I last researched the topic. If you know of a supported way of doing this on Linux I'd really love to know! It will allow me to finally make the program I wanted to make!

> As a result, the Linux equivalent to Everything is unable to find recently created files, erronously finds deleted files and has outdated metadata fo recently changed files.

Have you heard of https://www.man7.org/linux/man-pages/man7/fanotify.7.html ? This doesn't require watches on individual directories. Though:

> Calling fanotify_init() requires the CAP_SYS_ADMIN capability.

I agree with your frustrations around this limitation and think file watchers should do their best to support arbitrary file addition/removal in directories. That being said, the races do matter when you edit with "mammal speed" but use source control to merge/rebase commits that affect your watched directories. Especially in a team setting when multiple coworkers are working on the same directories, git can be blasting rapidly through a lot of file moves, deletions etc so it's not really an edge case.

Git is a good point. However, the only time I desire commands triggered by changes in source tree is during actual code writing to instantly test my changes. On the other hand stopping it during Git operations could harm my workflow. This could be easily solved by simple 1s debounce, with this hypothetical debounce filter

    inotifywait -rq -m -e close_write --format %f --exclude .git | grep '\.rules$' | debounce 1 | xargs ...

here you go: it looks at everything in a specific directory.

    `while inotifywait -q -q -r -e modify -e move -e create -e delete --exclude '[*~#]' $(mapi $wd $@); do`

wd is the working directory.

and mapi

    my $prefix=shift;
    foreach my $s (@ARGV) {
           print "$prefix/$s ";
    }

Adjust to taste.

it is _hax_, as it were. but it has worked well for me.

Have you tried Tup[1]?

[1] - http://gittup.org/tup/

Nodemon[1] is also good, normally Node centric but all parameters can be configured.

[1] https://nodemon.io/

> To install, get node.js, then from your terminal run:

There's no way I'm going to invoke node.js and associated `node_modules` with 1:10 code to junk ratio, for a any CLI scripting.

Reply on news.ycombinator.com