> The process was stuck in the kernel waiting for an IOCTL, so killing the process did nothing – even kill -9. It was going to stay that way until EBS was back to normal.
I did not know that can happen! Can anyone provide a further explanation/material for why this is the case?
It's possible to wait in a "interruptible" way, and many subsystems do. But that's hard, because it means that the resources you would otherwise be responsible for clean up need to be detected and cleaned up by someone else (or alternatively that someone else needs to pick up and finish what you started, or you need to be able to roll back a partially completed action, etc...). Userspace usually ignoes this problem (because the task got killed, right?), but in the kernel a leak is a huge bug.
So the default is that waiting in the kernel means that the kernel can't allow the process to be terminated (or otherwise be delivered a signal that would invalidate the stack). And the side effect is that userspace sees an unkillable process in the "D" state.
https://jira.mongodb.org/browse/SERVER-6178
http://www.noah.org/wiki/Kill_-9_does_not_work
http://stackoverflow.com/questions/223644/what-is-an-uninter...
The first is the bug I reported to 10gen about their MongoDB setup. Contained in the bug report is the set of commands I used to reproduce the issue so that you can see it yourself.
The second and third links are about processes in uninterruptible sleep, which is the state my MongoDB processes were in. It should give more detail than I'm about to give.
Linux processes all have a state (running, sleeping, etc). One of those states is "Uninterruptible Sleep". Only the kernel can change a process's state. Uninterruptible sleep ("D" in `top`) is a state in which the process will receive no interrupts (signals). Since the `kill` command is just a glorified signal sender, it ceases to work for those processes.