Hacker News Reader

The "but the OOM killer!" argument has been a disaster for Rust, and keeps derailing all design discussions.

• Linux is not the only OS in the world.

• Even on Linux you have containers/cgroups that can impose hard limits.

• Platforms without virtual memory are also an important target for Rust.

• On 32-bit platforms you can run out of address space before you run out of RAM.

• Regardless of what the OS does, the application may still want to impose its own internal limit (e.g. https://lib.rs/cap) to avoid being OOM-killed or swap death.

People keep telling me how Linux never runs out of memory, while I'm currently firefighting a torrent of coredumps caused by Rust's self-own on OOM that actually happens.

I generally love Rust, but its OOM handling is awful, and the "but the OOM killer" nonsense is to blame for stalling the absolutely critical fixes it urgently needs.

> but the OOM killer!" [..] disaster for Rust [...]

Only on HN discussions (the disaster part) ;=)

(Through it's is a broken discussion, OOM on Linux is broken, but it becomes increasingly "un-"broken and like you said other platform exists, too).

Anyway there is no disaster as:

Rust does support handling memory allocations gracefully, the allocator API defaults to this!!

Just various types defined in [lib]std (or precise [lib]alloc) default in there default methods to calling the (replaceable!) allocation error hook.

This is due to ergonomics, there is no sane/ergonomic way for "common non embedded/kernel programming or non special purpose use-cases" to make literally every method which potentially allocates return a result. And outside of such special purpose cases you can always recover from panics (i.e. outside of special purpose cases panic=abort is a anti-pattern).

Still, even in std, methods like `try_reserve` exists and are used, e.g. by serde deserializers to make sure to gracefully fail if the user tries to allocate a 10GiB array (because compressed data formats ;-) ).

BUT for the kernel all this doesn't matter as it is very unlikely it will use [lib]std or [lib]alloc. So they can default to just always implement their types in a way which doesn't panic on memory allocation.

And wrt. to panic in other (basically guaranteed to be a bug cases) panic=BUG() (ther kernel BUG() macro) can be done.

I'm more worried about the float/128bit integer thing he mentioned, as I don't know anything about it. I assume it's not just a miss understanding of overflow checks and other debug assertions the kernel likely will disable for non-debug builds?

> I'm more worried about the float/128bit integer thing he mentioned, as I don't know anything about it. I assume it's not just a miss understanding of overflow checks and other debug assertions the kernel likely will disable for non-debug builds?

Idk about 128bits integer, but the floating point question has been discussed on Rust's subreddit[1]:

> Normally, the kernel leaves the floating-point state of the CPU in whatever state the userspace process left it in, so that you can do a quick system call without having to save and restore all that state. It's possible to use floating point in the kernel with a great deal of care, but you have to notify the kernel that you're doing so, so that it can save the userspace floating-point state.

> So, it'd be helpful if by default Rust didn't allow use of floating-point, and then you could opt in to allowing it for specific code.

And from Linus[2]

> In other words: it's still very much a special case, and if the question was "can I just use FP in the kernel" then the answer is still a resounding NO, since other architectures may not support it AT ALL.

[1]: https://www.reddit.com/r/rust/comments/mqxr1a/rfc_rust_suppo...

[2]: (from the same reddit thread) https://ipfs.io/ipfs/QmdA5WkDNALetBn4iFeSepHjdLGJdxPBwZyY47i...

Well if that it all this could be solved by rust by having a "floating_point_usage" (or similar) lint which normally defaults to "allow" but for kernel devs defaults to "error" (which could be done through an option in the target spec, which as a side note is also what defines if you have hard,soft or no fp support). (there are probably better solution but this one should be the easiest).

There's actually a lint specifically intended for that already. `float_arithmetic`:

https://rust-lang.github.io/rust-clippy/master/#float_arithm...

# What it does

Checks for float arithmetic.

# Why is this bad

For some embedded systems or kernel development, it can be useful to rule out floating-point numbers.

Thanks, nice, but for the kernel use case it would need to be part of rustc.

Simply install clippy and call "clippy-driver" instead of "rustc", which will handle this. Clippy is largely the lints that haven't either made it into rustc or are fairly edge-case-y.

Why? The Rust-for-Linux team already made Clippy a requirement for Rust kernel code. For niche configuration that are really project-specific like this, it makes total sense to use external tooling to ensure the guidelines are respected, it doesn't need to be the compiler enforcing these guidelines.

The overcommit is not the only argument.

The second argument is that complex programs that need to handle OOM properly will be full of complex untested error paths.

How should one handle a OOM error If one can't allocate some temporary node during an algorithm and how to recover without leaving a corrupted state? That's not a question I want to answer for almost every function call.

That's why the rust standard library data structures don't handle OOM. To make it easier to use for the most common case. And for the cases where the OOM handling is required, one can simply use data structures for it, which is what Linux will do (problem solved)

Also compare that to the case of mutex poisoning which can return an error on each unlock, and which is now considered a mistake because in practice, nobody want to handle a poisoned mutex error.

I argue it's a rare-enough problem:

• Statistically, likelihood of failure is proportional to the allocation size. You'll most often fail to allocate a very large Vec, and then only need to allocate a few bytes for a string that says "error!". I'm using https://lib.rs/fallible_collections and just a few strategically placed try_reserve has brought down my coredumps by 99%.

• Panic on OOM would unwind, and unwinding tends to free memory rather than allocate anything. The standard library should reserve/preallocate enough memory to be able to start unwinding.

• Rust is good at avoiding temporary allocations and relies on the stack a lot. You don't have to use fancy error-handling libraries that collect backtrace on every error. Error handling with enums doesn't touch the heap at all.

• Even if everything fails, and you get OOM during OOM handling, that's fair, and still better than abort every time.

Note that Rust currently doesn't handle OOM by panicking. It unconditionally aborts the whole process. Libstd could switch to panics without API changes, and that would make OOM handling possible via catch_unwind.

> Panic on OOM would unwind,

OR abort the process.

Panics are for fatal irrecoverable errors, not for recoverable errors.

For example of the problem that the API would have if it wanted to handle OOM: Box, String or Vec, the most basic types in the rust alloc crate, couldn't implement Clone, because clone() can't fail, and you need to allocate memory to clone these data structures. As a result, lots of generic algorithms that relies on on their type to be clonable wouldn't work, and most user types could not #[derive(Clone)] anymore.

So, if I get this right, this means that the "just don't use the alloc crate" argument given somewhere else here is not practical ?

No. People that care about catching OOM failures will simply have to use other crates that have a more complex API, but that's what they want

I was just saying it is a right choice in the alloc crate, which is meant for user space programs, to simply panic on case of OOM, because otherwise the API would be more complex for everyone.

No not at all.

Because ...? I mean, from the perspective of an outsider reading that you "can't use strings" if there's a risk of allocation failure is a bit worrying.

Because you just don’t use the types in the alloc crate, or you use the (only partially implemented) APIs that return a Result<> instead of panicking on allocation failure.

For that kind of low-level work I’d probably do the former and implement my own data structures, which is what you do with no_std currently.

As others have said there’s nothing in Rust the language which assumes allocations always succeed. The standard library made the choice to panic on allocation failure, which was the right choice in order to make an ergonomic API that’s suitable for higher-level work. If you’re not using the standard library (which currently includes people on eg embedded targets) then you can handle allocation any way you like. And if you’re writing drivers for the kernel you’d assume you’re ok handling that.

GP was pointing out that the Clone trait doesn’t allow for returning a Result, so you can’t use that trait to clone anything that allocates. But that’s not really a big deal since you can just make your own TryClone that does.

Ok, thanks for the clarification.

Hows that' a problem?

Provide `TryClone`, there, problem solved.

I agree, it doesn't seem like these types can actually implement Clone if allocation can fail.

I agree that the OOM killer is frankly irrelevant in this discussion (and a bad idea in the first place IMO, cue the "airplane company deciding which passenger to throw out the plane" argument).

But on the other hand in my experience with C and C++, languages which do allow explicit OOM handling, most applications either just crash when that happen (the `xmalloc` route) or attempt to recover but often really can't do much.

It goes even beyond programming languages. Very few environments deal nicely with near-OOM conditions. My Linux desktop becomes effectively unusable if I start aggressively swapping. And technically at this point as long as I have swap I'm not literally out of memory.

My general point is not that I don't think Rust would benefit from allowing the developer to explicitly handle OOM conditions, it's more that the vast majority of applications are not written to gracefully degrade in these conditions anyway, and if the only handling you do is effectively `alert("out of memory!"); exit(1);` it's really not worth bothering with it.

On the other hand for the minority of applications that want to do something more meaningful on OOM you probably want to take the time to design some ergonomic system that won't litter the code with boilerplate, i.e. you probably don't want every single Vec and String and basically every single facet of the standard API that can allocate behind the scenes to return a `Result<_>`.

I'm sure there's something to do, but I think the Rust devs are right not to rush it given that there are so many other features left to implement.

> if the only handling you do is effectively `alert("out of memory!"); exit(1);` it's really not worth bothering with it.

I'd argue that even this is worth it, for most software. At least then you get a clean exit and an obvious problem. The alternative is to not detect it and let the code walk off into undefined behaviors and potentially subtle bugs that can harm data and/or break security.

Yeah, explicit OOM exit beats random NULL deref any day. The former can be analyzed by an L2 tech out of a logfile. The latter requires spinning up GDB to figure out where NULL came from.

In my experience the problem with the latter isn't even that it requires a skilled analyst & a debugger. The reality is that, in a multi-threaded program, by the time your process has crashed on a null dereference the offending call stack is probably long gone.

The faster you fail the more readily apparent the root cause will be from your logs/core dump/etc. For this to work though reboots (of the application/environment) need to be inexpensive, and you also need supervision. That supervision can either be at the OS level, like Linux's systemd, Solaris' SMF, Apple's launchd, etc., or at the application level like Erlang/OTP.

Yes, that's another great reason to fail promptly.

To be clear Rust will give you a clean exit in this situation (or at least, I assume that to be true, it would be a glaring issue if it didn't). On linux however since malloc usually can't fail you won't ever see that since the kernel will realize the issue on access and not on alloc, then the OOM killer will play russian roulette with your processes.

It would be really helpful, if the OOM killer should be adjustable to kill non-overcommitting programs last (kill any with overcommit first). Then programs could be slowly fixed/improved.

word

> For example, on Linux with overcommit (the default), even if the system is out of memory, malloc won't return null. It returns a pointer that's not null, such that your if(ptr == nullptr) will act as if everything is "ok",

Not always. The default overcommit policy will reject certain allocations that are too big. Also, you can change the overcommit policy if you need it.

Also, overcommit policy not rejecting allocations by default tends to be more useful than harmful. One example is forking processes: the memory space gets duplicated with copy-on-write, so if you didn't allow overcommit you could end up with processes that couldn't be forked due to that behavior. Not to mention that most programs don't use all the memory they have assigned, so without overcommit you'd have a lot of problems with RAM underusage.

> So I find it extremely ironic for Linus to argue that practical programming languages are hard to use for environments that must handle OOM errors, when they are championing one of the major platforms that makes handling OOM useless by default.

Kernel and user space programming are very, very different. A lot of developers use languages with garbage collection without issues, but that's not acceptable in kernel programming. The fact that the platform provides X feature does not mean it has to be developed including X feature. In this case it's very clear. Even if they used overcommit in kernel code, Linux does not always overcommit memory, so you'd still need to manage the OOM case in kernel properly.

So, basically you are saying "Rust is mostly used as an application programming language anyways, so let's not pester those users with the overhead of OOM handling"?

Fair enough. But this would mean that Rust is (factually) a systems language in the same sense that Go was initially declared to be a systems language.

If I look at the Rust community (I give it a try from time to time), I would totally agree with your point, that most users would be pestered by this overhead. I see mostly CLI tools, "web apps" and the kind.

Furthermore, I just can't shake off the feeling that the embedded (bare metal, 16 or 32 bit architectures) or (in-house) kernel crowd will always be 2nd class citizens.

    I keep saying this, but the obvious fix is for the Linux kernel to use overcommit internally 
    just like they expect user space to do.

I'm not a kernel developer, but aren't you (maybe) asking for a bit too much? You ask the Linux kernel devs to change the kernel in a significant way, just so you can use Rust for writing drivers in a way you write user space applications.

> So, basically you are saying "Rust is mostly used as an application programming language anyways, so let's not pester those users with the overhead of OOM handling"?

No. What I am saying is that Rust intended to be a systems programming language that could handle OOM properly everywhere, but that got a lot of opposition because some operating systems, mainly Linux, make it impossible for programs to handle OOM _at all_, so adding proper OOM support to Rust would mean that every Rust Linux programmer would be paying for a feature (in ergonomics, etc.) that they cannot use _at all_.

That's the irony.

Linux design has made it not worth it for new programming languages to be designed to handle OOM properly, because that's impossible to do in OSes like Linux.

People have been complaining about this for user space programs for the last 20 years, and the Linux kernel stand was "that's a feature, not a bug".

But now that they are confronted with it, you see Linus writing stuff like "that's not acceptable".

That's a huge double standard IMO.

I don't think you can use "double standard" as a derogatory term when you're comparing the needs of kernelspace code and userspace code.

...plus, they're already planning to write their own `alloc` replacement if for no other reason that they need to support API features of the kernel allocator that are absent from the userspace allocator, like GFP flags:

https://github.com/Rust-for-Linux/linux/issues/2#issuecommen...

> I don't think you can use "double standard" as a derogatory term when you're comparing the needs of kernelspace code and userspace code.

You argue that the needs of kernel space and user space differ when handling OOM errors.

If that were true, one wouldn't have had to patch the standard library with the `try_...` methods to try to support OOM handling in user space as an afterthought.

That request came from firefox, where a user in userspace can click on a link pointing to a 1 Tb large, e.g., image file, that would kill the browser if it can't handle OOM properly.

So yes, it is a double standard, because both user space and kernel space have millions of valid reasons for wanting to handle OOM errors, but Linux and in particular Linus Torvalds motto here is "do as I say, not as I do".

It is also extremely ironic that this Linux kernel policy that makes it impossible for user space to handle OOM, now causes Linus to cry about it when the kernel wants to start using tools develop for userspace in the kernel.

They are directly responsible for this outcome.

Rust 1.0 was explicitly intended to be a Minimum Viable Product, containing only the work they felt they couldn't revise later without breaking their API stability promise.

As such, they started with the versions of allocating APIs that are most commonly desired... the ones equivalent to how array[index] will either get what you want or panic if you're out of bounds. The try_ allocation APIs which are equivalent to .get on a container then came later.

A web browser, while a good program to stress the capabilities of a language's design to make sure it can meet demand, is not a typical program.

No it's about "by default" not pestering users.

Rust has all the tools needed to gracefully handle memory allocation failure.

Yes, the tools are a bit limit wrt. usages in the standard library where you mainly can use them to handle "known to potentially fail big" allocations but not every single small allocation.

But enter no_std and it's basically all your choice, which is also the default thing to use for embedded/bar-metal and I count the kernel as embedded/bar-metal. (It's default because the std library types are tuned for the most common use-cases including web-server and user-space system programming, in which you always can have panic on OOM, and error kernel as a much more ergonomic pattern then explicitly returning a result on every thing which potentially allocates).

Through without question the discussion around it is a mess.

You can't solve memory allocation problems by turning off overcommit and handling alloc errors in rust. You don't get Null, you get panic. That's a problem for the kernel, even if you argue it's not a problem elsewhere. Start killing drivers is nuts, oh look my screen went blank, and three HDs stopped working. Damn where is my swap.

> Start killing drivers is nuts,

I don't disagree, just saying that its as nuts as killing random user-space programs on OOM.

"The web browser misbehaved, lets kill the pulseaudio daemon, and if that doesn't fix it, lets kill WiFi, and if that doesn't fix it, lets kill vim, and... "

FFS if some app tried to allocate too much memory just let the app know, or kill that app, but don't start randomly killing user-space processes.

While I agree that OOMKiller is pretty mad, it's also important to note that the shared nature of memory means that the app which will die when the system is out of memory is anyway random. Even without overcommit, you can get into the situation of "the web browser is occupying 99% of RAM to open YouTube, and now pulseaudio tried to allocate another 50 bytes that the system just doesn't have, so pulseaudio should handle this, and then the WiFi manager will be the next to need 100 bytes and be refused etc".

Even worse, Linux by design stalls to a complete crawl when memory almost runs out, even with 0 swap and a fast disk, as it will first start swapping out code pages before giving up. Which means that every time a process is context switched, its code ends up first needing to be read from disk, the worst possible kind of cache thrashing imaginable.

Somehow a LRU-algorithm makes the most sense, if only Linux and humans would have a common definition of what "recently used" means here. And if applications had safe-shutdown and restart on demand mechanisms. On a high level of abstraction this is how a lot of VMs deal with things, such as tabs in your browser or open apps on your smartphone.

You are talking about userspace. Linus is talking about kernel.

You can change the Linux memory overcommit policy using sysctls if you need it.

die on run out of memory is a bad policy for a userspace server. Seen some hairy production incidents with that, basically solution is piss off customers until they stop coming back.

action on run out of memory should be finish up currently running work and hold off new stuff until you have resources, not die and start to reprocess the thundering herd. When you run out of memory last thing you want to do is empty your hot caches.

You’re out of memory. Every possible cache has already been freed.

Unless your app is faking it’s own caching, in which case it might be contributing to the problem.

It's quite possible that an app is holding onto data that can be flushed, at the expense of future database lookups, session negotiations, etc. Without a proper mechanism in the kernel it is hard to manage memory pressure.

If that would be important you would disable overcommit in Linux. If you didn't do that, it's probably because it's good tradeoff that saves resource.