Hacker News Reader

Comment by JoshTriplett | Hacker News Reader

This makes no sense. With the OOM killer disabled, the kernel shouldn't hang when memory allocation fails; it should return -ENOMEM. If something isn't handling that correctly, it needs fixing. In particular, if something in the kernel doesn't handle that correctly and propagate -ENOMEM back to userspace, it needs fixing.

View on HN

This is not about memory allocation. OOM occurs when too much of the allocated memory is being used. For example, forking happens with copy-on-write memory, so if I have a 3G process on a 4G machine, I can fork it 10 times without problems. It's only when those ten processes each start to write to their memory that the physical memory usage will quickly become too much to handle. In the scenario in the blog post, the OOM killer is disabled, and instead Linux can only prevent the memory from being accessed to avoid things getting worse. I would agree that this setup does sound like it needs fixing, but it's what we got in exchange for having cheap process forks.

Ah, thanks for the clarification; you're right, there's no way around that, short of blocking forks and speculative allocations unless there's enough memory to back them. That's theoretically possible, but extremely harsh, given the common case of fork/exec for instance.

I think it would be worthwhile to find a way around it, but it would require all applications to be conservative about their allocations. For example, instead of forking, sharing memory between processes could become something that has to be explicitly requested. It's probably never going to happen, given that these primitives such as fork and malloc are so ingrained, but when you consider what Linux is being used for nowadays, from mission critical servers to satellites, it might make sense to make things more robust at the expense of compatibility.

>For example, instead of forking, sharing memory between processes could become something that has to be explicitly requested A fine grained version of fork, clone, clone already exists. But it's not just fork, call stacks are also part of this problem.

How are stacks part of the problem? Given tail-call optimization stacks shouldn't grow unbounded, right?

I didn't see this until now, the problem is that when you spawn a new thread space for the stack needs to be allocated. Since the stack can't be reallocated easily in C/C++ it also needs to be "big enough". The current implementation is to allocate several megabytes of stack for each thread, since memory accounting isn't strict this doesn't create problems because unused stack doesn't really take up space. If you start accounting memory you will also have to be stricter with allocating stacks for new threads, limiting yourself to just a page or two wherever possible. This is not an easy task.

They shouldn't, but they do tend to have a large quantity of memory backing them that gets added to when grown into. The kernel uses 4k, 8k, or 16k stacks; a quick test on a Linux x86-64 system suggests that userspace has 8M stacks by default.

Hackish test code to recurse infinitely and print the stack pointer until a segfault:

    #include <stdio.h>

    static inline unsigned long get_sp(void)
    {
        unsigned long ret;
        __asm__("mov %%rsp, %0" : "=r" (ret));
        return ret;
    }

    void main(void)
    {
        printf("%lu\n", get_sp());
        main();
    }

Running this and comparing the first and last values shows a stack size of about 8M.

I do not understand. Where is the memory backing /proc/pid/cmdline and who in the container has a COW copy of it? If nobody, how can reading it cause more allocations in the container?

COW due to forking was just an example of how use not allocation leads to problems. I got the impression that if the container does not solve its own OOM condition, the kernel simply resorts to blocking all memory access. Perhaps blocking reads is not strictly necessary, as they do not have to increase memory use, but on the other hand it's possible that some of the memory already had to be discarded due to the OOM condition so it could have become inaccessible.

It may make sense for other things in the OOMed memcg to wait for the OOM killer. Otherwise the OOM condition is likely to crash everything (since nothing handles malloc failures correctly) instead of just crashing the OOMed process.

ISTM the bug is that things outside the memcg are waiting instead of getting ENOMEM.

Reply on news.ycombinator.com