Comment by eesmith | Hacker News Reader

Comment by eesmith | Hacker News Reader

Parent Full thread

How is it then that my stock KUbuntu system won't let me allocate 50,000,000,000 bytes?

  $ uname -a
  Linux boxcar 7.0.0-31-generic #31~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Aug 10 09:38:02 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
  $ cat tmp.c
  #include <stdio.h>
  #include <stdlib.h>
  
  int main() {
  char *s = malloc(50000000000ULL);
    if (s == NULL) {
      printf("Boo, hoo!\n");
    } else {
      printf("Look at all that memory!\n");
    }
    return 0;
  }
  $ cc tmp.c
  $ ./a.out
  Boo, hoo!

As to the lack of robustness of most software, that's a fact. But, for example, Daniel Stenberg of curl fame is a developer of robust software who does not appreciate how Rust handles out-of-memory errors makes it impossible to implement libcurl in the way he expects a library to work. See https://www.youtube.com/watch?v=HFH2vZRTKrA&t=2080s from 4.5 years ago as an example.

To get back to the essay, I can easily understand why someone who has Rust as their first-and-only systems language, and therefore expects abort-when-out-of-memory, will not immediately consider how C allows a different approach to how to handle that condition, but instead will try to replicate Rust's behavior in C.

"I can program FORTRAN in any language." :)

> How is it then that my stock KUbuntu system won't let me allocate 50,000,000,000 bytes?

Assuming that it has the default overcommit setting, it's based on a heuristic. Currently, that heuristic is something like "is this single allocation larger than all of physical memory + swap", if I remember correctly.

However, most real software does not try to allocate like this. It allocates reasonably small chunks of memory regularly. So again, in realistic conditions, you generally won't see a failure.

> Daniel Stenberg of curl fame is a developer of robust software who does not appreciate how Rust handles out-of-memory errors makes it impossible to implement libcurl in the way he expects a library to work

I am not re-watching this video right now, and of course have a lot of respect for Stenberg, but either he or you misrepresents the situation here: allocation is purely a library concern in Rust, not a language one, and so you can absolutely write your own code that allocates memory however you'd like. This is a concern about Rust's ecosystem, not about the language itself. It is surely not impossible to do this, embedded folks write Rust that doesn't even have an allocator present regularly, let alone infallibly allocate.

> 4.5 years ago

As I mentioned elsewhere in this thread, the standard library types are gaining methods that return Result instead of panic, and have had them for a long time, though they've been nightly only, since it's a pretty niche case.

My software does work with large blocks of contiguous data, where malloc can fail in exactly this normal way.

For one example, my library uses memory-mapped access, which is great for files which push the memory bounds. However, using memory-mapped files, especially over a networked file system, are slower than simply using RAM, so I have an option to read the entire contents into memory. I know the file size, I allocate space, and read in the data. If malloc fails, I report the failure, and let the caller figure out how to resolve things.

I don't think my software is all that special. A well-written library is not supposed to cause the entire program to panic. At the very least, the library doesn't know how to clean up all of the resources opened by the rest of the system. (Especially in a polyglot program where Python with a finally: cleanup calls C++ with a RAII object, then calls Rust.)

Concerning "I am not re-watching this video right now", I linked to the specific timestamp where he talked about it, so you wouldn't need to re-watch the entire video. I did that because the only text version I found was from 2020, at https://github.com/hyperium/hyper/issues/2265#issuecomment-6... , and I wanted something more recent than that.

> and so you can absolutely write your own code that allocates memory however you'd like

The linked-to essay uses new_uninit_slice in Rust, compares it to malloc in C, points out the issues should malloc fail, and how it needs something like an exit() to handle malloc failures.

My observation is that this would be an appropriate time to point out a philosophical difference between the Rust approach, where the default is for Rust to handle OOM errors, and the C approach where you are in charge of checking and handling malloc failures, and how some C library developers prefer different behavior than what new_uninit_slice provides.

Your suggestion that you can absolutely write your own Rust code this purpose also holds for the C code. You can write your own malloc wrapper function which does the check-and-exit() for you, if that fits your needs, then only call "malloc_or_die()".

> I don't think my software is all that special.

I think that regularly requesting a single chunk of memory that is likely to be larger than total RAM + total swap in the desktop context is unusual, but it's true this is an opinion.

> The linked-to essay uses new_uninit_slice in Rust

If you wanted to check the allocation, you'd use try_new_uninit_slice, which is the same thing but returning a Result instead of panic-ing.

> where the default is for Rust to handle OOM errors,

Again, it's worth splitting out the language from the library and or culture here: Rust itself never allocates, and so never OOMs.

> also holds for the C code.

I don't mean to imply Rust is better than C in this regard, only that "you can't check for OOM in Rust and therefore it's impossible to build the library how I want" is untrue.

My belief is that "C for Rust Programmers" should not be "how to write Rust in C" but "How to write C, for people who are Rust Programmers". The expectations for a production quality C library appear to be different than for a production quality Rust library, and I think one of those differences is memory allocation.

While I am certainly ignorant of Rust, my overall issue is that this statement:

"In order to avoid unhelpful segfaults, the program should check for a null pointer and gracefully exit if one is found:"

The following example shows that "gracefully exit" means a perror() and exit() call, which is not appropriate for a C library.

I presumed this is because the author is someone with Rust experience, and who wants to replicate the expected Rust behavior in C. I think the better alternative is to explain that a well-written C library can have different expectations.

I'll add that the essay has a footnote saying "The corresponding Rust function is not idiomatic, and can only correctly handle ASCII text. If I were writing a production version of this function, it would be a single line of code: forward.chars().rev().collect::<String>()."

I see that production version doesn't handle allocation failures either.

My follow-on presumption is that most Rust libraries use this sort of idiomatic production code, where allocation failures simply panic.

I could well be wrong, but this essay certainly doesn't clearly show that I am wrong in my belief.

That's fine. My aim is to clarify Rust's design and choices. I don't have strong opinions about what should be in the post.

> I'll add that the essay has a footnote saying

Honestly I have an entirely different beef with this footnote: `chars` is for codepoints, not really "characters", because what a "character" is in Unicode is very complicated.

> My follow-on presumption is that most Rust libraries use this sort of idiomatic production code, where allocation failures simply panic.

Yes, I wouldn't argue that most Rust code uses these try_ variants, because of the whole overcommit thing. But in contexts where it does matter, people do. The expectations are different based on context. Lots of Rust code expects (and enforces) no allocations at all, but that doesn't mean I'd claim that all Rust code does.

I also think it's fine to want to hold libraries, C or Rust, to a higher standard with regards to allocation failure, but in practice, most people (in both languages) do not find the juice worth the squeeze, and so don't really do it.

> How is it then that my stock KUbuntu system won't let me allocate 50,000,000,000 bytes?

I can't speak to the specific implementations, however servicing an allocation request has multiple fallible steps: reserve address space, then confirm/acquire/defer the physical memory backing the allocation.

My understanding is that overcommit allows deferring assigning physical memory to the allocation, however it could still fail to find a chunk of address space to service the allocation request.

Sure.

For context, steveklabnik suggested that malloc will virtually never fail on basically every Linux system, because of overcommit being the default.

I showed a trivial example of malloc failing, when trying to allocate more space than on my machine.

Another example is when using resource limits. I've modified my code to malloc only 500M bytes, the set a virtual limit of 500000KiB, which works, then 40000KiB, which doesn't

  $ grep malloc tmp.c
    char *s = malloc(500000000ULL);
  $ cc tmp.c
  $ ulimit -S -v 500000
  $ ./a.out
  Look at all that memory!
  $ ulimit -S -v 400000
  $ ./a.out
  Boo, hoo!

I clearly disagree with steveklabnik's because it's easy to demonstrate cases where malloc fails on Linux-based machines.

I learned programming from a tradition without overcommit; even though one failure mode is reported synchronously, I can't trust a success to mean I won't fail later.

"It's hard to handle correctly for beginners" is a reason for beginners to ignore a return value. It's not a reason for the system to (by default) not do the bookkeeping to know this will (likely) fail.

Sparse mapping & reserving address space then deliberately faulting in pages is a good & advanced technique, but it shouldn't share interface with the beginner/default API.

Reply on news.ycombinator.com