A pair of Linux kernel modules using Rust
lwn.net
lwn.net
This is the 5'th SubscriberLink over the last week on HN: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
Please, folks, subscribe, and let Corbet put the subscriber links here himself. The LWN weekly editions are definitely worth every penny.
Edit: I am paying LWN since 2012 and intend to keep paying for as long as there is LWN.
It's definitely worth subscribing if you have the means (and the low cost of subscriptions suggests a lot of readers here have the means).
Just to show support for corbet and his team. He could keep the articles paywalled forever, but is making them free after a week. Becoming a paying member is our way to say thank you for his fair attitude (and keeping ads at minimum level).
I work at totaly different field in IT, most articles are just "wow, that's interesting" for me and - I still keep LWN's subscriptionm and it is the only one I have.
> Must I subscribe to read the Weekly Edition?
> No, the Weekly Edition becomes freely available to all readers one week after its publication.
All of LWN's paywalled articles become free for everyone to view after a week. During the week that articles are paywalled any subscriber can share currently paywalled content by creating a SubscriberLink.
LWN produces some of the best highly technical Linux and Linux adjacent content on the web.
---
Are paywalls ok?
It's ok to post stories from sites with paywalls that have workarounds.
In comments, it's ok to ask how to read an article and to help other users do so. But please don't post complaints about paywalls. Those are off topic. More here.
---
I agree with one of the other posters, we should support quality journalism, and lwn definitely is.
Its the first post to gain any meaningful traction
> Please, folks, subscribe, and let Corbet put the subscriber links here himself. The LWN weekly editions are definitely worth every penny.
According to corbet its totally fine if every now and then a subscriber link pops up on HN: https://news.ycombinator.com/item?id=1966033
> As long as the posting of subscriber links in places like this is occasional, I believe it serves as good marketing for LWN - indeed, every now and then, I even do it myself. We just hope that people realize that we run nine feature articles every week, all of which are instantly accessible to LWN subscribers.
The LWN FAQ also considers it no problem: https://lwn.net/op/FAQ.lwn#slinks
> Where is it appropriate to post a subscriber link?
> Almost anywhere. Private mail, messages to project mailing lists, and blog entries are all appropriate. As long as people do not use subscriber links as a way to defeat our attempts to gain subscribers, we are happy to see them shared.
I think posting this on HN does not defeat the purpose of gaining subscribers, but supports them. At least for me it worked. After reading some LWN articles on HN I decided to become a paying subscriber.
Telling people they are not allowed to post such links here actually IMHO hurts LWN.
It's clear that the intention is to allow for sharing (although there is the question of "at what scale?").
Then there's the page itself:
> Welcome to LWN.net
> The following subscription-only content has been made available to you by an LWN subscriber. Thousands of subscribers depend on LWN for the best news from the Linux and free software communities. If you enjoy this article, please consider subscribing to LWN. Thank you for visiting LWN.net!
The people who run the site seem to have less of a problem with sharing its content than you do.
There was a big regression in compile times in the linux 2.2 to 2.4 time frame (the kernel grew a lot of functionality) but i don’t recall it getting much more than a little grumbling.
That was in the time of the Linux from Scratch book and Gentoo stage 0 builds. Some people don’t mind overnight compiles if it means they can specify just exactly the right combo of compiler flags to eeek out the max performance of their cpu.
I very much doubt that such large build times would go over well in the Linux kernel community.
That probably will never compile especially quickly. I'm not sure how sensitive kernel devs are to clean build times. Presumably doing a non-incremental build is relatively rare?
Can you link the code? Proc macros are not inherently slow to compile, but most proc macros pull in the `syn` and `quote` crates for convenience, which are fairly heavyweight (and then most people experience proc macros most commonly via Serde, which is doing a ton of work on its own). In simple cases you can forego most of that work, and some libraries like https://github.com/Manishearth/absolution exist as lightweight alternatives in these cases. Depending on what the Asahi driver needs it could also benefit from something like this (although without seeing the code I'm not sure why it would need proc macros when it sounds like conditional compilation via `cfg` would suffice there).
But You are certainly right in that external dependencies won’t be an issue.
However, I do see a custom `vtable` macro in there, and indeed macros can be a concern for compilation speed if they're not implemented with compiler-friendliness in mind. But other than that, I'd say that file shouldn't represent a problem for the compiler.
Getting CI up to speed is a completely different (awful) topic.
It seems like you could usually just synthesize the boxing needed. So why would you ever leave that choice in the library's hands, which has no idea how it will be used?
If by that you mean writing things from scratch instead of pulling a dependency that does it already, it sounds a bit counterintuitive.
The best dependency trees are as short and wide as possible. It means fewer changes with each upgrade, fewer chances of conflicts between them, and generally more-parallel compilation and better build caching.
Which is true in every language, but every community seems to need to relearn it from scratch.
For clarification:
> /proc/config.gz isn't available in Debian, because the config is provided in /boot/config-*, no need for the in-memory variant
Correct, 9p file servers can live in user-space. At some point you have to mount the fs which on lunix/unix is done in-kernel.
In plan 9 the kernel is just a VFS that lets you build arbitrary file trees from collections of mounts and binds. On-disk file systems like FAT and EXT are served via user-space programs which open the disk partition (served as a file by sd(3)) and translates the 9p messages to EXT, FAT, etc operations. File servers can also provide synthetic files, e.g. an arduino can serve its io bits and/or internal variables over 9p (ninepea on github written by echoine.)
Since its a VFS all the way down and data moves via two way pipes mapped to fd's you can boot and pull root from ANY 9p server. Don't care if its CWFS, HJFS, FAT, EXT, SMB, SSHFS, NFS, etc. Kernel don't care as long as it sees files through a 9p connection over whatever 2-way medium you are using including rs232. You cant touch that level of flexibility with other operating systems.
Of course, if you use an alternative executor there are large swaths of the (non-std) async ecosystem that are not available to you (thinking of anything like relying on a tokio::timer::Delay as an example). But as you point out Rust in the Linux kernel already doesn't rely on the std, so they wouldn't even begin to imagine to use an off the shelf crate and expect it to work unmodified for their use case. I just think it is worth pointing out that this is a constraint we currently have.
In a C module it would be undefined behaviour leading to kernel panic or weird bugs, in a Rust userspace binary it would be a panic that terminates the process with an error message, what happens here? Would it be a kernel panic as well? or is there a possibility of handling it?
EDIT: Look at replies. "Linus considers panics acceptable in some cases".
One I ran into (in the sense of read about, not experienced) was if I flatten a Vec of arrays, it's theoretically possible that the flattened structure has too many items in it to represent as a machine word integer. If this happens the flatten operation will panic.
This can't happen in a language like C (or C++) because their smallest type has size 1, so all the arrays can't possibly be bigger in total size than the amount of memory, that's nonsense. But Rust has two smaller sizes than this. The relevant one here is the Zero Size Type, Rust has no problem with the idea of an array of empty tuples, such an array could have say, a billion empty tuples in it, yet on a 32-bit system it just needs 4 bytes (to remember how many empty tuples are in it).
We can see that flattening a Vec of arrays of empty tuples is a pretty wild thing to choose to do, and nevertheless even if we do it, it only panics when the total amount of empty tuples won't fit in the integers of our native word size. But the function could be asked to do this, and so it might panic.
[ You might be wondering how can there be two sizes smaller than C's single byte types in Rust. The answer is the Never type ! and its pseudonym Infallible. The Never type is Empty, no values of this type are possible, so not only does Rust never need to store this type, it doesn't even need to emit machine code to handle this type - the code could never run. This makes sense in Generic programming, we can write Generic error handling code, but it evaporates when the error type was Infallible ]
Most likely you would have to use .get() which returns an Option rather than [] array index which panics.
So accessing an array out of bound will have a runtime check that will call the panic handler, and that panic handler calls BUG() which means kernel panic.
But I would assume it would be a kernel panic. It definitely won't be UB.
If an array and a length can be passed to a Rust module, I can just lie about it's size and, unless there is a runtime bound check (which can be slow), I guess bad things can happen too.
So, it's not a kernel panic, but it's extremely bad, which seems appropriate because your code is definitely wrong. If you're not sure whether the index is correct you can use get() or get_mut() to have an Option, which will be None if your index was wrong (or of course you could ask about the length of the array since Rust remembers how long arrays are).
https://elixir.bootlin.com/linux/latest/source/include/asm-g...
I guess that's quite drastic for a checked out of bound access, when there's no actual memory safety issue and the compiler can simply return an error from the function, or do something else less drastic.
It's impossible to claim there's "no actual memory safety issue" when a program's invariants have been broken: all bets are off at that point.
let foo = [0_u8, 1, 2];
foo[0].unwrap(); // really?
If you "know" the index is in bounds, you can get_unchecked. Otherwise you should get. Either would be a sane choice for the index operator.
But note his problem also happens with integer division.
In Rust, a[x] on an array or vec is really a roughly a shortand for a.get(x).unwrap() (with a different error message)
Likewise, a / b on integers is a kind of a shortand for a.checked_div(b).unwrap()
The thing is, if the index ever is out of bounds, or if the denominator is zero, the program has a bug, 100% of time. And if you catch a bug using an assertion there is seldom anything better than interrupting the execution (the only thing I can think of is restarting the program or the subsystem). If you continue execution past a programming error, you may sometimes corrupt data structures or introduce bizarre, hard to debug situations.
Doing a pattern match on a.get(x) doesn't help because if it's ever None (and your program logic expects that x is in bounds) then you are kind of forced to bail.
The downside here is that we aren't catching this bug at compile time. And it's true that sometimes we can rewrite the program to not have an indexing operation, usually using iterators (eliding the bounds check will make the program run faster, too). But in general this is not possible, at least not without bringing formal methods. But that's what tests are for, to ensure the correctness of stuff type errors can't catch.
Now, there are some crates like https://github.com/dtolnay/no-panic or https://github.com/facebookexperimental/MIRAI that will check that your code is panic free. The first one is based on the fact that llvm optimizations can often remove dead code and thus remove the panic from a[x] or a / b - if it doesn't, then compilation fails. The second one employs formal methods to mathematically prove that there is no panic. I guess those techniques will eventually be ported to the kernel even if panics happen differently there (by hooking on the BUG mechanism or whatever)
When the underlying _runtime's_ invariants have been broken, not when the program invariants have been broken. i.e. you can recover from almost everything save for a VM error in a VM language like Java, since there's no way for the program to mess up the VM's data structures in a way that they cannot be brought back to a defined state.
let innocent_var = operation_that_panics_but_returns_something_random_instead();
unsafe {
do_something_assuming_validity(innocent_var);
}... I wonder how many lines of code the Plan9 (C) server takes? (much less I suspect)
If the C ends up hand rolling something Rust just has built-in, that adds up pretty quickly. For example Rust's arrays, strings etc. know how big they are, C's do not and so you need extra code to re-calculate or to explicitly store and retrieve those sizes. On the other hand it may be tempting to express a Rust type as having methods defined over it, rather than as you must in C using only free functions throughout, there's also a Rust discipline of always deriving common traits e.g. Clone, Debug and Eq, when they're appropriate, as a convenience, that's one line extra in typical style.
You can think of the Plan9 kernel as a 9P multiplexer. 9P in, 9P out. The closest matching thing would be e.g. a 9P-to-ext4-in-a-file translator -- but it's always "9P server talking to $FOO" (or "9P server that makes files up on the fly").
Most of the complexity of this Linux 9P server would be in translating between the unrelated worlds of 9P and Linux VFS. A more apt comparison is the size of this 9P server vs an in-kernel NFS server -- they both translate something unrelated to Linux VFS.
More seriously: many different security improvements are filtering into the kernel idea-by-idea, insofar as folks working on kernel security do the actual work of making them fit in with the kernel, and/or coming up with better alternatives.
(Oddly, grsec itself has expressed opposition to Rust in the kernel, I think on the grounds that their custom GCC plugins can't cross the language boundary. Now that there's work on supporting Rust in GCC itself, I'm not sure if that objection still applies.)
Some of these people are so obnoxious...
Vaguely descriptive attacks that serve no purpose other than religious wars. When I see statements like that I see another person who thinks it’s simply fashionable to hate popular languages because they heard someone like Torvalds, Thompson, or Stallman say it. If you have a fundamental explanation like they do, then it's at least reasonable, disregarding the irrelevant context of a Linux kernel forum thread. But I rarely see that, instead I find mutter like "cpp is bloated, unstable, overcomplicated crap with ugly syntax, that's why I use rust, or nim, or any other pretty new thing that was conceived 35+ years later".
For instance:
> c++ is so brittle that a gentle breeze could cause a cacophony of soundness issues
Okay, how so, anything about C++ or just C in general?
In languages as powerful as Rust and C++ you can express programs which fall into three categories, two of these aren't very interesting, the programs which are obviously valid, and the programs which are obviously nonsense. We know what to do with these programs, the former should result in emitting correct machine code, the latter should cause a diagnostic (error message). The problem is the last group, programs whose validity is difficult to discern. The bigger and more complicated your software the more likely it may end up in this last group.
In C++ the third category are treated as valid. In the ISO standard this is achieved by having clauses which declare that in certain cases the program is "Ill-formed, no diagnostic required" which means the standard can't tell you what happens, but you won't necessarily get an error message, your compiler may spit out a program that does... something. Maybe it does what you expected, and maybe it doesn't, the standard asks nothing more.
In Rust these are treated as invalid. If you try hard enough (or cheat and Google for one) you can write Rust programs which you can reason through why they should work but the compiler says no. You get an error message explaining why the compiler won't accept this program.
Now, if Rust's compiler doesn't like your program, you can rewrite it so that it's valid. Once you do that, which is often very easy - you're definitely in that first category of correct programs, hooray. The program might well do something you didn't intend, the compiler isn't a mind reader and has no idea that you meant to write "Fnord" in that text output and "Ford" is a typo, but it's definitely a valid Rust program.
In the C++ case we can't tell. Maybe our program is the ravings of a lunatic, the compiler isn't obliged to mention that and we are in blissful ignorance. This also provides little impetus for the standard's authors or compiler vendors to reduce the size of the third category of programs, after all they seem to compile just fine.
Obviously it'd be excellent in theory to completely eliminate the third category. Unfortunately Rice's Theorem says we cannot do that.
Inherently, or in the compiler implementations you've seen?
There's some discussion here https://stackoverflow.com/questions/7237963/a-c-implementati...
Now of course you could build a compiler which rejects valid programs anyway and is thus not complying with the ISO document. Arguably some modes of popular C++ compilers are exactly that today. But now we're straying from the ISO C++ standard. And I'm pretty sure I didn't see a modern C++ compiler which reliably puts all the "No diagnostic required" stuff into the "We generate a diagnostic and reject the program" category even with a flag but if you know of one I'm interested.
I called both of these (and compile times) when people first tried to get rust on the linux kernel. Maybe it's better to acknowledge these problems before you start instead of beating your head? I'm no psychic and I know I wasn't the only one who called these
What's unacknowledged? They're doing exploratory work. That often involves developing under known and expected limitations to understand the problem space and have something to test with when progress is made towards addressing limitations.
Here in the Linux Plumbers Conference talk regarding the NVMe driver (the same one discussed in the OP), the benchmarks show nearly identical performance to the existing implementation. At the end of the talk the co-author of the NVMe spec (Matthew Wilcox) made a point to stand up and comment with how unexpectedly impressed he is with the performance: "I was not expecting to see these performance numbers, they are amazing." (at this timestamp: https://youtu.be/Xw9pKeJ-4Bw?t=9486 ).
EDIT: here's the timestamp of the benchmarks themselves: https://youtu.be/Xw9pKeJ-4Bw?t=8626
The inclusion of Rust in the kernel is one of these ongoing bits of work which isn't ready for mainline yet, but has still generated a lot of interesting discussion in the past on whether Rust is suitable at all, or whether it's suitable in it's default form, or if it is suitable then what is the best way to start introducing it, or, or, etc...
And this is another step along that (long, convuluted) path. The proposed 9P implementation specifically - to me at least.
If you mostly care about the Linux kernel insofar as what features it can offer you right now then, sure, this might not seem that interesting. And that's entirely fair - there are plenty of other technologies out there that I have that level of interest in.
But for the people who take a deep interest in the upcoming work and future direction of Linux, I think it's quite interesting.
I'd agree that the article probably is a bit too niche, or forward-looking, for a generic tech website/magazine, and doesn't have a huge target audience. But for a focussed site like LWN, I think it fits in pretty well, and matches the audience that LWN targets. It's definitely the kind of article that I keep my subscription for.