Hacker News Reader

Comment by tptacek | Hacker News Reader

Everything will probably be just fine if you kill-9 something. If a program fails permanently and dramatically when it's kill-9'd, you should "remove it from your filesystem", because it can't handle other, unavoidable abrupt failures either.

View on HN

> Everything will probably be just fine if you kill-9 something. If a program fails permanently and dramatically when it's kill-9'd, you should "remove it from your filesystem", because it can't handle other, unavoidable abrupt failures either.

There is a lot of middle ground between failing "permanently and dramatically" and shutting down cleanly.

What about a program that can recover after an abrupt termination, but only after a time-consuming recovery from a journal? What about a program that can recover after an abrupt termination, but only after you manually remove a lock file? These are cases where it's not "just fine" to use SIGKILL, but not so bad as to warrant removing the program.

Generally your life will be easier if you don't take the sledgehammer approach to killing processes, and at least try a non-SIGKILL first.

Your reply is the real answer to the linked question. SIGTERM serves a real purpose, and almost always it works to end the process cleanly and promptly.

If a process doesn't respond reasonably to SIGTERM, then you should consider removing it from your filesystem.

It might not be that simple as ops folks often have to support specific versions of stuff on apps that use db-specific features. In that case, it might be worth a prof services contract with a db shop.

(I've supported clustered Oracle on AWS handing massive numbers of micropayment transactions and seen weird shit in prod where it "kind-of failed" according to our ops dba. Classes of survivable bugs vary from apply a vendor hotfix to edge cases not worth downtime.)

> If a process doesn't respond reasonably to SIGTERM, then you should consider removing it from your filesystem.

What's reasonably? MySQL/InnoDB will start consolidating the journal and buffer pool in preparation for a shutdown. On machines with a large amount of RAM allocated to the buffer pool that will take quite some time. Is that still reasonable?

Absolutely. Those are healthy parts of a clean shutdown, which is what SIGTERM was really asking for, isn't it?

InnoDB is designed to recover gracefully from a SIGKILL, too. The journal task is "saved" for later. That's why they tell you in DBA school, "don't use MyISAM tables. They're not safe." Because if they received a SIGKILL or power outage at the wrong moment, writes could have been lost. Amiright?

What you don't expect is for a process that receives SIGTERM to fail hard, and require an expensive journal recovery or suffer some unrecoverable data loss as a result.

Well, actually: Recovering from a SIGKILL will (or at least used to) take much longer at startup than shutting down with SIGTERM, so it could handle that better. I still agree with you that it's reasonable, but others may hold a different opinion.

Heh - you're saying that the SIGKILL recovery on start is cheaper than the SIGTERM cleanup on quit. OK. I'll agree, that's strange. One would probably have expected to have to dot and cross the same number of I's and T's either way, and not for SIGKILL recovery to actually be faster.

I didn't know that.

No, actually I wanted to say the opposite: Killing the process with SIGKILL and then doing recovery at startup is much more expensive than letting the process shut down properly with SIGTERM.

This paper measures the time difference between clean and unclean reboot across various systems. Another important point is that many servers never shut down intentionally, making unclean shutdown the norm in lots of deployments.

https://www.usenix.org/legacy/events/hotos03/tech/full_paper...

Thanks for the hard numbers. That also makes sense.

It should be easy to rationalize that SIGKILL recovery is slower than SIGTERM shutdown for such a database. If your in-memory cache is empty, it won't be providing any speedups, right? Thus you'll need to go back to disk for everything.

Unfortunately the signals are badly named. Last time I explained to someone the difference between SIGTERM and SIGKILL. He was baffled.

He: You mean SIGKILL isn't the signal for graceful termination? And it isn't sent by default when invoking `kill <PID>`?

Me: Nope. Why do you think that?

He: Well, the command is called `kill`.

Me: ...good point.

It's even worse than that -- kill is just a command that sends signals to a program, some of which happen to (gracefully or not) stop the program. You can even define your own!

It's the difference between "kill it" and "kill KILL it". :)

If we use analogy it would be like forcing the process to kill itself (http://www.merriam-webster.com/dictionary/terminate) vs killing it (http://www.merriam-webster.com/dictionary/kill).

"If a process doesn't respond reasonably to SIGTERM, then you should consider removing it from your filesystem."

That's an overreaction. I've seen programs refuse to respond to SIGKILL for reasons out of their control, like getting stuck in an NFS transaction or other "unusual" filesystem and the kernel being unable to process the SIGKILL. "Listing a directory" is not exactly a crazy thing to do.

And no wiggling out by saying "well, that's the kernel, not the process", the process never gets a chance to "handle" SIGKILL, so it can't really screw it up, either. Arguably, all such failures are the kernel; the kernel should not expose any sequence of calls that causes SIGKILL to fail, and over time they tend to be fixed (I haven't seen this on my modern Linux machine in a long time, even playing with some funny stuff), but it has happened and will probably continue to happen as new stuff comes out.

>And no wiggling out by saying "well, that's the kernel, not the process", the process never gets a chance to "handle" SIGKILL, so it can't really screw it up, either.

Technically, you are correct- the process doesn't even know it's been SIGKILL'd. However, there are other things it could do to gracefully handle the scenario upon next start. Is there an already-existing lock file? Prompt for its removal. An already existing PID file? Again, ask what to do. Include tools to fix records that may have been left in an inconsistent state. So on and so forth.

That said, I've never had to do a SIGKILL. However, I still expect my programs to sanely recover from power outages, clumsy interns, RAID failures, and other "acts of god" that may suddenly cause a program to end before it has a chance to clean up after itself. It's part of making robust programs.

> If a process doesn't respond reasonably to SIGTERM, then you should consider removing it from your filesystem.

I currently have a vim process that is running in the background and not responding to SIGTERM (so I'll probably SIGKILL it). Does this mean I need to find a new editor?

I was also thinking that criteria might prune the filesystem a bit too much. Alternately, if you're a vim hater, "Another good reason to get rid of vim!"

EMACS responds to SIGTERM. Just sayin... :-)

Vim responds to SIGTERM while in the foreground, but if I suspend it (e.g. C-z), it does not respond. If I send a SIGTERM to a suspended Vim process, it does respond immediately after I foreground the process.

Should I stop using NFS, or should I remove the Linux kernel from my filesystem?

Yes!

*BSD with ZFS.

I agree for the most part. There are a few exceptions, however, and no-one seems to have noted/listed them. Most notably, when a process is 'kill -9'ed, some shared system resources stay open. Most notably of those are the named semaphores. This will sometimes cause the process to fail when it is reopened. You can use ipcs and ipcrm to remove those.

One thing I always liked about couchdb, aside from its other merits or faults, is that it was designed to be a crash only service. Meaning that the normal, correct way to shut it down is just an abrupt halt of the process. This means that functionally there's no difference between a normal shutdown and a crash or a killed process. This is quite valuable for a service where data integrity is important.

This is a feature the tux3 filesystem also has, and I hope it will be merged into Linux when it is stable.

> If a program fails permanently and dramatically when it's kill-9'd

There are several kinds of bad state that can be left over:

  (1) space leaks (disk, memory (sysv shared memory))
  (2) inconsistent data (data / config files)
  (3) locks/semaphores (used to protect (2) from happening in normal operation)

(We're talking about those that are under the control of the program itself, not the control of another program or the kernel (the latter can and generally will be cleaned up automatically).)

Programs can be written to clean those up when they are restarted, which is the reason for the saying that if some program does not do that, then it's not worth being left installed.

But it takes a certain amount of work writing code to recover from an abandoned lock and the possibly inconsistent state that has been left over, so much so that even some wide-spread system libraries don't do it. Of course you're free to remove those system libraries from your system, but you may have to write a replacement.

Also, in the case of such libraries, it may happen that you lock out other programs (that also happen to use the same libraries) than the one you SIGKILL'ed, and you may not be aware of the source for the issue.

In particular I'm thinking of ALSA. If you SIGKILL a program while it uses [a "hw:" device in] ALSA, it leaves behind a semaphore that prevents other programs from accessing the same device indefinitely (it seems there's no recovery code in the ALSA libraries). This robbed me of quite some time to figure out; when I did, I wrote a utility to clean up the semaphores explicitely[1]. Every now and then it happens that I need to reach out for it.

[1] https://github.com/pflanze/chj-bin/blob/master/alsafix

This is absolutely, technically true, but sometimes it's just so much easier to write something that doesn't handle being kill -9ed gracefully...

hangs head in shame

Well, as you can't catch -9 (SIGKILL), any task in your program that shouldn't be interrupted... will be interrupted. That's generally a good rule, but programs aren't usually SIGKILLED. It is SIGTERM the one you're supposed to take into account.

Permanent and dramatic failure caused by SIGKILL presents a candidate for deletion, but it depends on your definitions of permanent and dramatic.

It wouldn't be unexpected for a SIGKILLed process to fail to flush a cache, or to exit with persistent state ...inconsistent. That can be permanent (changes lost) and dramatic.

You can wrap operations in transactional overhead to greatly reduce the chance of data loss, but you can't eliminate it entirely without assistance from peers.

kill -9 is the equivalent of unplugging the machine in the middle of operations. It might be fine, and with careful design it might be fine almost every time.

Why should "transactional overhead" not eliminate it entirely?

And why "might be fine almost every time" instead of "will always be fine"?

Insert as many 9s you want in 99.9%, it'll still be a finite amount, so it's not "aways".

That doesn't change when you add peers, though, so it's kinda besides the point. Oh, and a normal shutdown doesn't prevent flipped bits due to cosmic rays either ...

I think some mathematicians would disagree with you here :-)

I think that's rather harsh, MySQL/InnoDB can't handle kill -9. Granted, kill-9'ing a database process is bad karma :)

I think you may have reinforced tptaceks's point, actually. Many DBs can handle abrupt termination.

isn't that the whole point of journaling! Of course database can handle kill -9, they are designed to. That doesn't mean it's friendly to do that because you'll pay a price in redoing journals.

also -9 can leave ipc hanging so some ipcs might be needed.

Not that the database will get corrupted, but that's a good way to lose data.

A database should not lose any committed data on `kill -9` in the default configuration. This is why PostgreSQL waits on fsync on the write-ahead log before completing a commit. This can be disabled with synchronous_commit off, in which case you will indeed lose data on a crash.

Normaly, it's not nice to lose uncommited data.

I said it already elsewhere in this thread, but either the writer has other persistent storage, then it should just keep the data and retry later and nothing will be lost, or the writer doesn't have other storage, then you will also lose uncommitted data with a normal shutdown, as you can not even try to commit any data while a data base is down.

If a database cannot handle kill -9 then it cannot be considered transactional and shouldn't be used.

Unless you're talking about filesystems with specific support for atomicity, and databases with custom storage drivers for those filesystems, that's really not fair.

It would be far too easy to kill any program mid-write().

A database should not be corrupt if a kill -9 is issued and it failed mid write. That is a design criteria that cannot be compromised.

That is what journaling is for. A database that corrupts data when you kill -9 it is garbage. A database has to survive without corruption when power fails unexpectedly, and that's even harsher than kill -9 in terms of what can go wrong if your code is sloppy.

> Unless you're talking about filesystems with specific support for atomicity, and databases with custom storage drivers for those filesystems, that's really not fair.

Unless you're talking about a database not intended to be used in production, that's really not fair.

What if the power comes down? The UPS blows up? There's an earthquake and the ceiling comes down? Is it OK for a DBMS to be corrupt its data then, too?

> Unless you're talking about filesystems with specific support for atomicity, and databases with custom storage drivers for those filesystems, that's really not fair.

Yes it is. The DB should not unrecoverably corrupt itself in case of hard power loss, and a kill -9 is significanly less traumatic than a hard power loss (which includes PSU melting/explosions or UPS going down).

Databases must handle termination, that's right. But they may lose data doing that, and there's nothing wrong in that (it's even inevitable).

Actually, it can. Killing it with -9 prevents the process from writing the buffer pool etc. to disk, so that the the server needs to recover that state from the journal/log files. No data will be lost though. So you're basically trading time at shutdown vs. time at start-up.

> So you're basically trading time at shutdown vs. time at start-up.

I'm being a bit pedantic, but trading time at shutdown vs. time at start-up + potential data loss (if the journal was in the middle of being written while the -9 was sent, you probably lose the last journal entry).

If that counts as data loss, then a "normal shutdown" also causes data loss. When the journal write gets interrupted, the database client won't get a success return, so it should not delete the data it was trying to commit, and retry the transaction at a later time, so nothing is lost at all. When the client is some entity without reliable persistent state that can not store the data in order to commit it later, that entity will be unable to connect to the database once the "normal shutdown" is completed, and thus will face the exact same problem, as the journal entry is "lost" before the write even starts.

> If that counts as data loss, then a "normal shutdown" also causes data loss.

I disagree. If a journal write was in progress on SIGTERM, you either fail or complete the write, returning the result to the client. In either event, you should end up with a clean journal on SIGTERM, assuming your implementation was sane (i.e. how I'd write it). If the client chooses to ignore the failure, that's not on the server and that's the whole point.

A normal shutdown should never require data loss. Somehow you're conflating SIGTERM with SIGKILL and I'm not understanding why.

A client that relies on receiving an explicit failure response is broken. When the server is killed by the OOM killer or when power of the server fails or the network connection between the client and the server fails (for too long) while the commit is in progress, the client also doesn't get an explicit response, and thus also doesn't know whether the commit succeeded or not, so it has to deal with that case anyhow - and if it doesn't lose data in that case, it won't lose data on SIGKILL either.

And the whole point of a journal is that it doesn't need to be clean. If your database requires the journal to be clean on startup as a condition for its correctness, then the journal is useless, it could just as well just require the data itself to be consistent. The only reason why there is a journal is so that correctness is not affected when writes are interrupted as any point.

Also, for that matter, a SIGTERM might be able to guarantee that the journal is clean, but it would be highly broken if it tried to guarantee that the client gets to know about the result, as that could take an arbitrary amount of time that might be dependent on the behaviour of remote systems, which would be terrible shutdown behaviour indeed.

Really, the only reason why a database should even try and catch SIGTERM is when checkpointing on shutdown is a lot cheaper than log recovery on startup - other than that, it only makes the code more complicated without providing any benefits.

Of course it can. Have you ever tried it?

Reply on news.ycombinator.com