Comment by sigil | Hacker News Reader

Comment by sigil | Hacker News Reader

Traps are neat but beware, they aren't completely reliable. You can't trap SIGKILL or SIGSTOP. In the article's examples, a SIGKILL would (1) leave temp directories around, (2) fail to restart a service, and (3) leave an expensive AWS resource running.

Remember that SIGKILL isn't always the result of a human typing `kill -9` either: the Linux OOM killer sends it; all unixes potentially send it during shutdown and runlevel switching; programs like timeout(1) send it as a last resort.

Here are some other ways to approach the 3 examples:

1) Avoid temp files and directories if you can. Sometimes you can't, but anecdotally I come across LOTS of shell scripts that create a temp file when they could have used a pipe. Bonus: pipes are fast.

2) Insuring a service comes back up after maintenance: use a process supervisor with automatic restarts, and have the service script grab a startup lockfile first thing. Use a blocking flock(1) or setlock(8) and discard the lock fd immediately afterwards. To bring the service down for maintenance, grab the startup lockfile, stop the service, then do your thing. Once your maintenance script exits -- through any means, including SIGKILL -- the kernel automatically releases the lock and the hitherto blocked service continues starting up.

3) Capping expensive resources: if the EC2 instance truly is temporary, why not impose a timeout and police all such instances out-of-band, with an alarm? The article is right that omissions of this kind can be $$$. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

> Traps are neat but beware, they aren't completely reliable. You can't trap SIGKILL or SIGSTOP.

A long time ago, there actually was a sneaky way on some Unixes to trap SIGKILL. If a program was being run under ptrace then any signal would pause the program and alert the program that was doing the trace--even SIGKILL.

So I made a program that I named "sh" and carefully made to have the same memory size as /bin/sh, that just forked and exec'ed another program of mine under ptrace. The other program was named "superman". Whenever my fake sh received notification that "superman" had received a signal, it would write the number of the signal into a variable in superman's address space, and then make it so "superman" continued but with the signal changed to SIGINT. The SIGINT handler in "superman" would would check that variable to see the real signal, and print an appropriate smart remark.

I started this running, then went to the head system admin/system programmer and told him something was wrong and I couldn't kill my program. After seeing that ^C and ^\ did nothing useful he logged into another terminal, became root, found "superman" with ps, and did a kill -9.

The look on his face was priceless when "superman" just printed something like "SIGKILL is not strong enough to harm a Kryptonian!" and continued running.

I was a little sad when later Unixes made SIGKILL kill processes being traced.

I love this story. Thanks for sharing!

I suppose these days you could just write a kernel module, either bundling a shell language in there, or do something root-kit like with protecting the process...

Of course, it's worth noting that if you lose power or your computer is torn asunder by an act of God, then of course your exit trap will also not execute. Point of mentioning that, is to drive home the point that you can't ever fully rely on exit traps or mechanisms like exit traps (i.e. Go's defer, Python's try/finally...) so you must always still consider failure modes beyond that.

Mitigating failure modes beyond that can be challenging, but usually there's something you could do that will go further. For example with EC2 boxes you could find some mechanism to make the box 'temporary,' perhaps tag it as such, then have a program on another machine somewhere that reaps these boxes after 24 hours. (edit: Honestly, rereading the parent post, this is pretty much the same thing they recommended, but I didn't really realize it when I wrote this.)

It's still useful to use exit traps and friends, but as you mentioned, it's not a silver bullet, and you can make a lot more robust systems if you try to handle a layer or two more of failures.

SIGKILL is never received by an application, I believe SIGSTOP isn't either. Both of them are calls to the kernel for managing a process, and SIGKILL should be reserved for extrenuating circumstances (an application won't quit normally...). SIGKILL gives no application a chance to clean itself up, shell script or no.

This is hardly a special feature of Bash, it's just how the operating system works.

> This is hardly a special feature of Bash, it's just how the operating system works.

I never claimed otherwise!

Yes, all regularly scheduled processes should be able to recover gracefully from a forcibly interrupted state. Programmers ought to be thinking about "what happens if the power goes off right NOW"?

especially item 2. wrapping a critical section in a lock is great. I prefer to have the default state be "running", so if the maint script ends for any reason the lock is discarded. If the maint proc gets stuck you can see it's holding the fd on the lockfile, or you can put a watchdog on it (implementation depending on your supervisor).

i love these little bash gems, the only downside for me is I do feel like I'm stepping back into the 20th century whenever I do any heavy bash work...classic systems programming...

> I come across LOTS of shell scripts that create a temp file when they could have used a pipe.

Wouln't a pipe live on in the file system just as well (unless you're talking about things that can be done simply with a "|")?

re: 1) any way to use a (named) pipe to replace a directory besides flattening out the dir?

I wouldn't say so: any pipe only allows strictly linear FIFO data transfer, but anything you could call "directory-like" surely allows random access - at least jumping to the start of any of the contained files.

I think, if you wanted a truly self-tidying temporary directory... you might be able to mkdtemp(), immediately chdir() into it then unlink() it, and thereafter work on the files within by relative paths only - as the directory's link count would be zero, it should get automatically cleaned up when the process exits. (Well, unless your program gets killed between the mkdtemp() and rmdir(), of course.)

... alternatively, you might also be able to mkdtemp(), opendir() then rmdir() it, then work on the files in it with openat() & friends. I can't see that being viable in shell scripts, though - too little support for dirfd-relative file operations.

Not sure if either idea would actually be allowed in practice, mind - I haven't tested them, either might fail at the rmdir() stage with EBUSY.

You can only rmdir an empty directory and then after that it's gone and can't have files created in it.

Provided you didn't need concurrent access, what you could do is create a temp file, unlink it, then treat the contents of the now-anonymous temp file as e.g. a zip file or a tar file or similar. But that strikes me as one of those "time to rethink the life choices that led you to this moment" situations rather than a remotely good idea.

I actually tried this idea when I read their initial comment. Unfortunately (And this is obvious when you realize it) you can't `unlink` or `rmdir` a non-empty directory. Also, though `rmdir` can work while you still have an open reference to the directory, once you do a `rmdir` you are not allowed to add any new files to the directory, making it essentially useless.

I think the best way would just to make sure you mount a `tmpfs` somewhere and make a directory on that (With something like your PID in the name). Then have some supervisor program periodically remove all directories with PIDs that are not longer running (Along with having the programs themselves clean it up if they have the opportunity). That's probably the best you can get, I don't think there's anyway to make the directory get automatically cleaned-up by the kernel.

> once you do a `rmdir` you are not allowed to add any new files to the directory

Ah, go figure. I had a feeling you'd run into something like that.

As for your tmpfs idea... you could do something with autofs to get automatic cleanup.

Yeah, it's unfortunate, it'd be pretty cool if you could pull something off like that. And conceptually it's not much different then what you can do with files already.

And autofs sounds like it could do it. I think the simplest implementation wouldn't be very complicated though, you could probably do it with a bash script that cross-checks folder names with running processes every 5 minutes or so. Assuming most programs clean-up after themselves, it only needs to handle programs that do an abnormal exit like SIGKILL. Of course, if that script has problems you'd be in for a bad time.

Reply on news.ycombinator.com