Hacker News Reader

Comment by ParrotyError | Hacker News Reader

The OOM killer was a misfeature when it was designed. Why is it still in the kernel? Solaris solved this problem 20 years ago.

View on HN

Overcommit is desirable because of the fork/exec model of creating processes; if a big one wants to spawn children, without overcommit you need to be able to recommit its VM size, regardless of how lightweight the desired child is.

Sadly there is no good (& simple) solution under most Unixes (that's a problem that only affects long running and big processes, but those are often the main applications of some embedded systems). I'm not really found of vfork now that we use threads more often (well, at least on Linux it seems other threads are not suspended, but it seems that this is not a Posix guarantee)... posix_spawn could be a solution if implemented with some dedicated kernel support (or always implemented through well-behaved existing ones).

> if a big one wants to spawn children

Should this be considered poor behavior or design?

It's been quite a while since I wrote explicit fork/ exec code, but wouldn't a better approach be to have a small master process that spawns off the necessary children and then either links them up or mediates communication?

I mean, on a unix-like system, init is ultimately the spawner of everything else and it's not a particularly large process.

Yes on Unix it can be interesting, if needing fork/exec to spawn children, to pre spawn an helper child process that will do just that. But that would just be because the fork/exec model is insane: with a posix_spawn (or CreateProcess) centric approach, you don't have to add that extra layer, and you have very few drawbacks. Also, that will not help you when trying to use libraries that attempt to spawn on their own.

As for init being small, I leave you the responsibility of your words, now that we live in a systemd world :p

You can do this:

vm.overcommit_memory=2

And then the kernel will no longer do overcommit.

Indeed. But the fact that the kernel does it at all is mysterious, and that on most systems the misfeature is enabled by default...

The reason it is there is that the C memory allocation model tends to ask for rather more memory than the program actually needs. This leads to inefficiency if all that unused memory is backed by physical memory. So this overcommit trick was created to allow other processes to 'steal pages' from processes that have allocated memory but have decided not to use them for now. In practice this works reasonably well, but there are some nasty edge cases when one or more of those programs wake up and then decide to actually use the memory they think they already have.

For systems where deterministic behavior is more important than flexibility (for me: all of them, for others: their choice) you're better off disabling overcommit.

I don't think there is no such thing as a "C memory allocation model". There are various ways of requesting memory from the OS. But not requesting anything in advance is going to be very slow in any setting that requires a context switch or virtual memory.

A typical memory manager will scale up the size of the requests to the OS whenever it runs out of space.

Pardon my ignorance: How did Solaris solve this?

I can't remember but I did sit in on a presentation about 15 years ago where they explained it. I lent the notes to a senior developer and never got them back.

It doesn't have an OOM killer. Even more remarkably a call to allocate memory can't fail, but it may not return either. When Solaris (well, SmartOS in my case) runs completely out of memory, all hell breaks loose.

I just read http://unix.derkeiler.com/Newsgroups/comp.unix.solaris/2008-... and it seems that they did not solve any problem at all; they don't allow overcommit, that's all. But they do not do anything else to compensate for the disadvantages this implies (this also have advantages, but not only)

You can do that by tuning Linux to have this behavior if you like it. In a Unix system, I don't thing this is the most desirable behavior in the general case...

Not every system has virtual memory, in particular Android uses the OOM killer pretty aggressively to manage transient applications.

In that case OOM killer is a pretty nice solution compared to the alternatives.

Reply on news.ycombinator.com