Why can't we just save Rust data-structures in a memory-mapped file? That would make saving and loading much more efficient, and it would also make programming much more convenient.
Rust's in memory data structures don't have any meaning on their own, they only have meaning in the context of the running program.
This is, in fact, the whole point of the New Type Idiom. If I have two types Minutes(u32) and Years(u32) those are represented in memory the same way, an unsigned 32-bit integer, just four bytes. But because Rust is statically Type Checked, everywhere the programs knows which is which, you can't assign a Minutes to a Years, or a Years to a Minutes, for the same reason you can't assign "Hacker News" to an f64.
But this means a 32-bit value in your hypothetical file could be either a Minutes or a Years, and a program loading that file needs to know a priori which it is. Once you start coming up with rules about how you'd do that, you're inventing a serialisation format, and you're exactly back to the problem where your original question arose.
I've used the crate `savefile` for this in the past (https://docs.rs/savefile/latest/savefile/), which also supports migrating data between versions. So if you have a bunch of on-disk data of a struct and you add a new property, you can handle migrating the data on the fly to the new struct structure.
I'm also using `savefile` for saving HN items (comments and stories) to disk in my own HN application: https://codeberg.org/ditzes/ditzes/src/branch/master/src-tau... and it's very quick to load stuff from disk, haven't hit any bottlenecks really
Example: https://ditzes.com/item/35640002 (SpaceX Starship rocket explodes minutes after launch from Texas) has 1223 comments which are individually loaded from disk, takes around 100ms to load from disk, which considering how inefficient and unoptimized the code is, I'm pretty happy with.
Which is exactly what savefile does.
It is certainly possible to store data structures directly inside files (I think MS Word did this to save documents), but indeed you have to think carefully about type safety but also data alignment, offsetting and such, and perhaps cryptographic signatures to make it secure when needed, which is what library/language designers need to think about to make it easier for programmers to use this stuff. Sure, it won't be adequate in all situations, but that's not what I'm asking for.
Once you solve the various edge cases, this is basically what flatbuffers and capnproto are
It is possible to do explicitly with `#[repr(C)]` in Rust. This is still fragile:
- `usize` and `isize` are not guaranteed to be portable between machines.
- pointers (including vtables in trait objects) are not portable between processes
- structs and enums may change in source code
- file descriptors, mutexes/futexes and any kind of external resources in general are not portable between processes. Thread-local variables are not even portable between threads.
In short: memory layout in general depends on a large amount of ephemeral compiler metadata. It is possible to constrain memory layout to something portable (protobufs, flatbuffers, capnproto, thrift, etc), but this seems more like a serialization format, not an efficient memory layout for a compiled language.
I'd say this is a good thing to avoid ossification of memory layouts: if you need to unmarshal something, it is better to mark this explicitly. Although the C ABI can be convenient, it constrains optimizations that compilers are allowed to do.
And there are the challenges with pointers and data structures others have mentioned. And supporting old files in the event of a Rust ABI change becomes incredibly hard.
While Windows enforces it, most Unix systems don't support it at all. Back when Linux had it (removed in 2021), it had to be enabled when a volume was mounted, and that was not default nor was it supported for all file system types.
Without it, even if you mmap() the file with the MAP_PRIVATE flag, portions of the file could be overwritten by other programs before you read them.
There's a reason most serialization formats are either self-describing, have external schemas (json), or offload state to a specific state store engine.