Rust in the Linux Kernel: Just the Beginning
memorysafety.org
memorysafety.org
I want to be able to actually compile my software if I wish so. This is becoming increasingly difficult. Rust is adding to the problem.
We should aspire to make it as fast but in the short term a slower compiler in exchange for less CVEs and random buffer overrun crashes seems like a reasonable trade off to me.
Distributions such as Fedora offer build infrastructure[1] that you can use to compile packages to use in your system for testing if you feel your local hardware isn't powerful enough.
(Same reason why C with typedefs is slower to compile than plain C, why C++ is slower, etc)
(that and cargo dependencies, etc - also C compilers have some +30yrs of optimizations)
A full Rust kernel might be neat. A compromised Rust driver inside of a C kernel is never going to be the default choice.
The actual driver code is pretty readable, see https://lwn.net/Articles/863459/.
May I interest you in Redox? https://www.redox-os.org/
Obviously very different than Linux.
There's been multiple passes at making the Rust compiler faster. It'll happen. There's a GCC-based Rust implementation in the works too.
> The main difference between Rust and other languages is that it does some more (but not all [2]) safety checks at compile time rather than at runtime. It also allows you to avoid GC, but does not provide you any memory safety over GC. Rust's borrow checker allows you to statically prove that references are live [3]; a GC simply avoids deallocating any memory that has a live reference to it (on the other hand, a GC can ensure that references remain live even where this is hard or impossible to prove statically). The end result is the same with respect to memory safety (the reason some people want to avoid GC is for performance reasons, not memory safety).
> The main difference between ARC and Nim GCs is that ARC is fully deterministic - the compiler automatically injects destructors when it deems that some variable (a string, sequence, reference, or something else) is no longer needed. In this sense, it’s similar to C++ with its destructors (RAII). To illustrate, we can use Nim’s expandArc introspection (will be available in Nim 1.4).
> This shows one of the main ARC features: scope-based memory management. A scope is a separate region of code in the program. Scope-based MM means that the compiler will automatically insert destructor calls for any variables which need a destructor after the scope ends. Many Nim constructs introduce new scopes: procs, funcs, converters, methods, block statements and expressions, for and while loops, etc.
> ARC also has so-called hooks - special procedures that can be defined for types to override the default compiler behaviour when destroying/moving/copying the variable. These are particularly useful when you want to make custom semantics for your types, deal with low-level operations involving pointers, or do FFI.
You can always make it your own personal default with older versions (e.g. nim-1.6), by editing your $HOME/.config/nim/nim.cfg to say so or doing similar on per project/file basis.
Rust also uses the same mechanisms to get compile time thread safety with full memory sharing between threads. Does any other language that doesn't have a global lock, throwing away most of the advantage, have that? There are actual new and interesting advantages to the compile time ownership model.
A minor advantage on the age of microservices and OS IPC to shared external resources.
Sendable doesn't apply when those threads are accessing shared external resources.
If you want fully lock-free it might be possible to prove that with extensions like Liquid Haskell, via Linear types, and is definitely easier to prove with a theorem prover than it is, generally, for C++ code. Not sure about Rust though I realize quite a lot of its moving parts have already been formalized which is super cool.
There are reasons for using Rust but it's not the only game in town. And GC doesn't automatically mean pessimistic performance.
Rust is a huge, huge win for that reason. It allows for code written in a Lisp/OCaml/Nim style that's memory safe without any additional overhead.
Rust compilation is not intrinsically slow; better IR and more incremental compilation can and will improve it. Incremental compilation (only changed files, not crates) is 7 years in making though.
[1]: https://prev.rust-lang.org/en-US/faq.html#why-is-rustc-slow
Other reasons:
- suboptimal LLVM IR code, as you mentioned, and other tech debt
- the preferred strategy of monomorphising generics is fast at runtime, but slow to compile, using trait objects is faster
- the complex type system of course also plays a part
- LLVM itself is of course not optimized for Rust
I'm not sure it was a "bad decision" if it was something that can be addressed after the fact and allowed them to ship working software more quickly.
The vast majority of the popular projects I am aware of have this kind of technical debt, likely due to survivorship bias. Project teams that refuse to take on technical debt are rarely successful enough to become popular.
Maybe we are giving Rust developers computers that are too fast. ;-)
OTOH, it's always wise to design for the future, so it also makes sense to give Rust developers beefy server grade hardware so they can play with SIMD pipelines and all those extra cores and threads, because that's what workstations will be a couple years from now.
It is only a matter of time, until GCC and clang finally catch up.
Shouldn't this be enough? Or does it require explicitly including the precompiled header whereas VC++ has magic to make that transparent?
Ada, Delphi, OCaml, C#/F# (.NET Native / Native AOT), D, Nim,...
For everything else regarding concurrent access to shared data out of process, they are on the same foot.
Ada didn't took off, because of several reasons, price of compilers, on UNIX it was an additional SKU on top of the respective developers SDK that already offered C and C++ in the box, 1980's mainstream hardware wasn't able to cope with it, most OS vendors outside UNIX decide to migrate from their toolchains into C and C++, so again additional money on top of the OS SDK.
Early Ada compilers had expensive licensing and required expensive hardware to run.
By the time GNAT was added to GCC, C++ had already taken over most of the spaces that were not Ada exclusive (meaning safety critical / defense / aerospace niches where either Ada was once mandated or has thrived in despite the original high costs).
I believe the point the author is making is that other languages provide better safety than C and have faster compile times than Rust, therefore Rust should be able to improve its compile times.
Ada with SPARK 2014 formal proofs, for your other part of the remark.
What you may want to try to avoid is complex and non-deterministic GC, which makes it harder to reason about.
Except most of C# ecosystem relies on classes and GC. A lot of these problems are caused by overuse of GC-ed classes, and their ease of use.
C and C++ aren't immune to heap abuse as well.
You do on C# just like on them, think about data structures, and avoid heap during render loop
They still ran into limitations where basically "This would be a lot easier if we didn't have GC".
tr;dl — I’d really prefer the option to determine when I’d like GC, as opposed to dodging the collector to avoid performance hits, hot code can default to manual. Probably not a realistic ask, but it could work.
Fair enough. If SS14 used D maybe they wouldn't have these problems.
D and early Rust (pre 0.2, like alpha alpha) had that. Problem is you split your community in two. You get a version of "What GC-color is your function/lib?".
It's a tradeoff for some domains - allow no GC bypass you're going to run into nigh insurmountable performance cliff.
Allow GC as opt-in and you run into issue of splitting your APIs in two.
"This would be a lot easier if we didn't have malloc()/free()", basically.
In any case Sony, Nintendo and Microsoft have been doing graphics stuff with C# for their platforms, although they could have kept being pure C++.
Also we shouldn't silo ourselves into only C# as discussion point, Go, D, Nim, Swift, Eiffel are also possible examples.
Well, not really. It's just some libraries like YamlDotNet copy waaaay more than needed and it shows in serialization. They mostly minimized calling YamlDotNet, but true solution would be a zero copy parser.
Other issue was HashSet operation like Clear had huge impact. Think they replaced those with arrays.
Third issue was something about flecs and archetype ECS. I don't know if it was a jest, but they mentioned changing GC layout or rewriting GC.
Points is, they now face a steep performance cliff. The only way out of it is through sheer effort.
So no, it's not anti-GC religion. Some domains and GC really badly mix.
Came across this page, on the motivations of Gentoo, a few years later:
> Installing a working Linux box used to require over 550 man hours, learning a Nordic language, sacrificing a goat, wading through hundreds of pages of (purposely) inscrutable help files...Old-school Linux users were desperate to find a new way to feel superior.
- the Linux project is very likely to stick to simple, fast elements of Rust (based on the excellent approach of the Linux/Rust devs thus far)
- the more Rust is used, the more work will be done to improve its performance
- you can still build a kernel on a low-powered device... i've built kernels that took > 12 hours on, for example, PA-RISC boxes that were once regarded as beefy :-)
- most people don't (and shouldn't) compile their kernel, and by most I mean more than 99%
All I really want is to be able to compile my stuff without waiting overnight (or more).
Not only I woundn't say that most people shouldn't compile their kernel, I would say that most linux users* should do it at least once, so they can understand the power they have compared to closed-source operaring systems.
*with linux users I mean users that use linux as their main operating system, not people that do ssh once in a while or rarely boots their linux partition
a modular kernel with a custom inird (generated by the distro) is small enough for most.
so if you are into adventures or you are in the business of kernel development yes roll your own. anybody else is better served standing on the shoulders of a maintained binary distribution.
- everyone has better things to do than compile software they didn't write
- a good distro has probably tested it on a bunch of hardware, and hopefully signed it (or at least the packaging), so you know it was securely acquired and built
- you won't learn much at all about the Linux kernel by compiling it... you may learn a tiny introductory about about it by configuring it, but that's still not very much at all, really (it may seem like a lot when you don't know how to measure what you're learning)
- what you should learn from configuring and compiling a Linux kernel is that you don't ever want to be in a situation where you have to do it again (without a really spectacular reason, or being paid)
- if you're compiling a kernel because your boot partition is small... make it bigger, or don't have one at all. come on.
With every year, abandoning OSS and looking for non-computing hobbies gets more attractive.
I haven't checked the times, but if the borrow checker is really the slowest part, maybe making rust skip it is a valid approach for end users. Sounds like an interesting experiment.
Regardless of how fast one's machine is, I think having some compile caching infra like sscache should help improve time to compile.
[0] https://perf.rust-lang.org/compare.html?start=9be2f35a4c1ed1...
Relevant username?...
In my experience when compilation takes longer the actual delays are in the last steps, long after the compiler is done checking and printing warnings.
On an unrelated note, I also noticed a large amount of static string literals in the code can slow down compilation to a surprising degree.
So to bootstrap rust this way, you'd need to go GCC 4.2 -> GCC 13+ -> Rust
https://seclists.org/oss-sec/2022/q4/23
This is a catastrophic bug, that (after some work on developing an actual RCE) lets anybody within wifi range to get root on your laptop (or phone, or access point). And all it took is this one line:
https://git.kernel.org/pub/scm/linux/kernel/git/wireless/wir...
We've all been repeating the "1000 eyes - all bugs are shallow" mantra for far too long. This one was in the mainline for more than 3 years, and nobody noticed. How many more are lurking there?
memcpy(pos, mbssid + cpy_len, ((ie + ielen) - (mbssid + cpy_len)));You can turn on runtime overflow checking with the Rust compiler, but you can do that with gcc compiling C too.
I guess the answer is that you wouldn't do the memcpy like that in idiomatic Rust - you'd use some higher level construct that gives the compiler more chance to catch your errors. Could anyone comment on how this works in a case like this?
edit: Can't reply to repliers. Defined overflow doesn't help here. A checked_add() function could be used in the Linux kernel in C, just as easily as in Rust. Forcing checked_add() to be used in the Rust compiler would help, but would also have performance and readability impact, which is presumably why it is still being debated.
In Rust the u8 type wraps in default release builds, but panics on overflow in debug builds, but this is kernel code so it will definitely be built in release mode with wrapping.
Yes, that's the trick: it will be caught not in the u8 overflow, but either in the memcpy equivalent (which is "dst.copy_from_slice(src)"), or in the slice manipulation before it. What happens is that slices in Rust are represented by a "fat pointer", a pair of the starting address and the length, and both the "copy_from_slice" method and the index/index_mut operator check the bounds before doing the operation.
(You could do things using "unsafe" and raw pointers, which have only the starting address without the length, but in idiomatic Rust you'd use slices most of the time.)
DoS is the only publicly known exploit, right now. There is understanding that RCE is possible with additional specialist work.
Local packet injection is an implementation detail of the first public PoC. There are other implementations of the same exploit, that deliver packets over the air instead. Including one that runs on ESP32, and attacks unmodified Linux nearby:
Just leave a small board, a battery, and a solar panel on a roof or tree near your office. If RCE is possible with this (people think it is), this would be very valuable.
On a tree near my office, you'd gain access to our network, Microsoft's and a European central bank. Not bad at all.
- The Linux kernel and issuance .so files lack a "view source" button on compiled binaries. And even checking out the matching source, building a replacement binary, and diffing your local changes from the matching source is an arduous progress to setup per program/library from a tarball/Git tag, wait for the computer to finish, install dependency .so files globally, ensure symbols are present, ensure you can breakpoint static functions...
- Dynamic dispatch and generic code might help maintainers and code extensibility but (in my experience) definitely impede external eyeballs from understanding code.
To a smaller extent, netbsd has pkgsrc, and most BSD friends have something like that too.
While I agree that bug is serious, that "some" is doing pretty heavy lifting here. Is there an RCE for this bug?
* Overflow with the default operators is not UB in Rust, it will either panic or two's compliment wrap, depending on various things (including things you can set to choose this global behavior). This already prevents various issues.
* You can also explicitly choose to do various operations with whatever overflow semantics you want, as you mention with saturated_add and friends.
* Because indexing is bounds checked, where in some languages an overflowed integer would lead to incorrect indexing and therefore possible memory problems, you'll either get a panic or a logic bug, not a memory bug.
* If your integer isn't being used for indexing, you'll end up with some sort of error, but again, at worst a logical error, not a memory error.
It turns out that compiler speed makes development faster, keeps people interested in the language, and ultimately allows more iterations before release (which can be better for speed and safety than throwing in a bunch of extra compile steps).
Rust offers a better trade-off between compilation time and other parameters, than C++.
And let's admit even though Rust compilation speed could use some improvement, it's not terrible.
And yes, a substantial time is also spent for things like the borrow checker but no all of it.
An no, it is not a niche requirement. Short compile times are absolutely critical for developer productivity. One main reason Golang exists and got popular is that people got fed up with how slow C++ is to compile. Not to mention that most people on this earth are not as privileged as to have a beefy machine.
Can you back this up? I'm not a programming language expert, so if there's some common knowledge you're referring to, I'm not aware of it.
> most people on this earth are not as privileged
There's very little "privilege" you need to compile Rust. In my spare time, I develop mainly in Haskell on my ThinkPad X270 and it goes just fine. Building libraries takes time, yes, but you need to do it just once. And while it's building you can think with a piece of paper more — also privilege in a way.
Now, I wonder what is the most reasonable option: writting a naive and simple 'c11' compiler or a naive and simple rust compiler.
I wonder if somebody has done a "syntax complexity diff" between 'c11' and rust.
I know that linux is written in "gcc C", not 'c11'... so...
On the other end of the software stack, we have servo, mozilla web engine written in rust. What's up there? Still a drop of rust in a ocean of c++? (SDK included).
Because after years, if it is still impossible to run servo without c++ code, this is bad omens for kernel rust.
The idea here would be to compile in a way that just assumes everything is correct and either crashes catastrophically or produces invalid output otherwise. But in doing so, it should allow at least slightly faster compilation.
You could even remove the need for this "fast and loose" compiler to do any type inference by shipping pre-processed source with all types resolved. However I don't know if this addition would fit your needs if, e.g., your goal is to be able to compile from any given commit rather than only official releases.
At very least it could be an interesting experiment to discover what tradeoffs are possible.
Majority of the time is spent in LLVM, because rustc throws a ton of code at it to clean up. This is being addressed by MIR optimizations (rustc's built-in optimizer working on higher-level code) to remove costly abstractions before they become a pile of low-level code to eliminate.
So what you're saying is that you're opposed to millions of people having more secure software, and perhaps millions of dollars spared from breaches, because it makes your own occasional singular personal experience of compiling the software faster?
Or did you mean something else?
I know this because I've managed to add a form of RAII and borrow checking to portable C11, and C is known for being faster than Rust to compile. Imagine what happened if we made a language with that stuff built-in.
The funny thing is that C is also slower to compile than it could be because of headers.
So you can of course compile your Rust compiler. If you are used to compile clang or gcc, it's not that much of a hassle. And the benefits have already been shown. If you only want to compile Rust code, and not develop it, mrustc might also be a good choice for you (it doesn't implement borrowck, just what's needed for codegen).
Finally, if you don't want to use Rust drivers, you can simply configure them out and don't need to build Rust. It'll be quite a long while until Rust will arrive in the kernel outside of drivers (which tend to benefit most from Rust anyway).
If I had a choice between 50% CVEs and 2h per compile or 5' compiles, I'd take the former in a blink.
I followed the https://rustc-dev-guide.rust-lang.org/building/how-to-build-...
Then ran the build step "time ./x.py build -j 8" ... Build completed successfully in 0:36:53
real 36m53.398s user 254m52.720s sys 12m48.289s
Seems pretty reasonable considering it's not a particularly high end desktop from 2015. Seems like a cheap price to pay for increased reliability and security.
Fast compute at this point is quite literally the least expensive part of the equation. Machines will get faster. Compilers will get optimized.
We've spent decades optimizing the developer experience (compile times) at the expense of the rigor, robustness, and quality of our resulting product. I've been doing this for 30 years, and I can categorically say that I've spent FAR more time chasing NPE, OBO, and race condition bugs than I would have ever added to my build time with a slightly slower compiler.
itripn&
It seems, there is a wide group of developers which thinks that Rust is the best candidate as a kernel development language. If you see issues with that choice, now would be the time to propose an alternative and try to find momentum in the developer community supporting that alternative. While I also lack practical experience there, by all what I heard, ADA could be one. But I don't know how it exactly compares to Rust and what the trade offs are. But so far, no one has pushed for ADA as a possible kernel implementation language.
Does anyone know how much Google pays for this kind of stuff? And is it like a limited-in-time kind of contract or something similar?
Rust as a language to write kernel drivers for <POPULAR ARCHITECTURE> is a great achievement, but what about all the other architectures that only have a C compiler and do not want/can't to depend on LLVM?
What about the billions poured into LLVM that made writing an optimizing Rust compiler possible in reasonable time, that is still written in C++?
Lastly, I hope to be wrong on this, but watching at Google history being backed by them is, unfortunately, a course,
I guess the most critical part of a Linux Kernel is being able to compile it in the first place for the architecture you're using.
Virtually every platform/architecture out there provides a C compiler.
Can't answer on the CVE part, we have no data to discuss the matter in a meaningful way.
There's certainly hope, it doesn't mean data will prove us right.
We have
https://msrc-blog.microsoft.com/2019/07/22/why-rust-for-safe...
https://www.chromium.org/Home/chromium-security/memory-safet...
You can't compare the number of bugs in a 20 years old C software (C++ in the case of Chrome, which is another beast entirely) with wishful thinking.
The reality is that whatever data you want is pointless. Anyone who knows what they're talking about should be able to very easily conclude that Rust will have fewer memory safety issues than C. Arguing about that would be silly. I think to nearly the same degree that one can draw the conclusion that Rust code will have fewer CVEs than C. Generally speaking, of course. Individual projects can always be outliers.
Why?????
Just wait.
Obviously you can't compare 20 years of data with "a few papers"
It means you're comparing hard evidence with something that is theoretical.
Imagine this:
"Cancer has been one of the main causes of death for people around the World, but we have these few papers about this new drug that say that it can beat cancer, se we can say for sure that cancer will not be a main cause for death anymore, without even testing the new drug, because I've read it in these papers here, written by people who work at developing this new drug, so obviously we don't need no data!"
If anything, the liks at the studies posted here prove that memory safety issues went down over the years, even without Rust.
Because, obviously, when going fast to develop something, you cut many corners, then, when your software is stable enough, developers can focus on improving it.
95% of the bugs were memory safety issues doesn't really say anything.
If those 95% issues were all edge cases, does it really matters?
If 50% of that 95% was something like "in some cases the interface is rendered slightly incorrectly" does it really matter?
I am much more worried by the 0,1% of them that end up in an arbitrary code execution vulnerability or worse.
> The reality is that whatever data you want is pointless
Then why even bother?
If it's pointless, why are you even arguing with me?
You're right by design, I'm quite sure I am not.
> Anyone who knows what they're talking about should be able to very easily conclude that Rust will have fewer memory safety issues than C
Doesn't mean anything.
If you put people in bunkers alone you'll see a drop in homicide cases.
Doesn't mean it's better.
That's all I'm saying.
I'm not a zealot, not a fan, don't have a political agenda and not trying to convince anybody.
We don't have the data, that's all, if you have data then show it to me or you have to agree with me that we don't have any.
I guess you put Linus Torvalds in the group of people who don't know what they are talking about
Sorry if I trust him more than you.
---
And the *reality* is that there are no absolute guarantees. Ever. The "Rust is safe" is not some kind of absolute guarantee of code safety. Never has been. Anybody who believes that should probably re-take their kindergarten year, and stop believing in the Easter bunny and Santa Claus.
-- Linus Torvalds
I... just explained. You can't control for the variables.
> If it's pointless, why are you even arguing with me?
I'm not arguing with you. I made one comment to explain that the data you're asking for doesn't exist and if it can exist it is extremely expensive to generate. Generating expensive data like that to show something obvious is not something most companies care about.
> We don't have the data, that's all, if you have data then show it to me or you have to agree with me that we don't have any.
Yes, I'm saying that there is no good data.
> I guess you put Linus Torvalds in the group of people who don't know what they are talking about
Yes, I think Linus Torvalds is incompetent. That said, I think you're actually misunderstanding his statement there.
Therefore, for the Linux kernel specifically, I think the only concern is whether or not GCC remains supported in addition to LLVM - as GCC and Clang are, as far as I know, the only two compilers which actually can be used to build the kernel as we speak. There's work being done on both gcc-rs and rustc_codegen_gcc to allow using GCC as a Rust backend, meaning that all platforms currently supported by the Linux kernel should be eventually capable of being supported without porting a compiler backend.
I believe that was a feature.
non-GCC compiler, like Clang+LLVM, are (were?) considered to be open source but not Free Software (TM)
for some at least, and it was a goal in the first two decades of the Linux kernel community
TL;DR: LLVM and Clang allow non-free (proprietary) extensions
The Clang and LLVM developers reach different conclusions from ours because they do not share our values and goals. They object to the measures we have taken to defend freedom because they see the inconvenience of them and do not recognize (or don't care about) the need for them. I would guess they describe their work as "open source" and do not talk about freedom. They have been supported by Apple, the company which hates our freedom so much that its app store for the ithings _requires_ all apps to be nonfree. (*)
The nonfree compilers that are now based on LLVM prove that I was right -- that the danger was real. If I had "opened" up GCC code for use in nonfree combinations, that would not have prevented a defeat; rather, it would have caused that defeat to occur very soon.
For GCC to be replaced by another technically superior compiler that defended freedom equally well would cause me some personal regret, but I would rejoice for the community's advance. The existence of LLVM is a terrible setback for our community precisely because it is not copylefted and can be used as the basis for nonfree compilers -- so that all contribution to LLVM directly helps proprietary software as much as it helps us.
The cause of the setback is the existence of a non-copylefted compiler that therefore becomes the base for nonfree compilers. The identity of that compiler -- whether it be LLVM, GCC, or something else -- is a secondary detail. To make GCC available for such use would be throwing in the towel. If that enables GCC to "win", the victory would be hollow, because it would not be a victory for what really matters: users' freedom.
It was beneficial to Linux, the open source and free software, as in GPL licensed, kernel.
It was beneficial to its users.
It was beneficial to the free software movement.
It wasn't beneficial to corporations, probably.
Things in fact have changed when corporate sponsored interests raised around Linux as a money maker platform. And when Apple started its war against GPLv3 (suddenly GCC was not good anymore for them) .
> Who benefits from the Linux kernel not being able to be compiled by a certain compiler, and how?
Again: the fact that Linux was not being able to be compiled by a certain compiler doesn't seem to me it hindered its ability to become the most used platform in the server space (or one of the most used).
Any counter proof?
-----
As already stated: beneficial != convenient
Going to the doctor for regular checks is not convenient, but it's beneficial for your health.
Apple started to use Clang because GPLv3 was stricter than GPLv2.
That's why you can compile Linux with Clang, because Apple did not want to use GCC anymore!
That was a feature back in the day.
It forced corporations to adapt to Linux, instead of the opposite.
Apple doing open source feels more like PR than anything else (they do "maintain"/"employ the main dev of" cups if I am not mistaken).
And yes, open source is not enough anymore: we need "lean" open source, and that includes the SDK (excluding de facto the ultra-complex c++ and similar).
The real hard part is, once a piece of lean open source is mostly "done", it is to resist planned obsolescence.
That said, I am a "everything in risc-v assembly" (with interpreters of high level languages written in assembly) kind of guy.
do you have any evidence of that? and why are you quoting RMS while talking about development of the linux kernel?
Yeah, the fact that only few years ago Linux was finally buildable with Clang.
If you have any proof that there were other reasons besides "nobody was working on that and the Kernel was using GCC extensions not found anywhere else" I'll be willing to look at them
These debates have been going on since the mid 90s, I don't know why people still ask questions about them, it's ancient history, GCC was the de facto compiler for Linux because Linus decided so and everybody else followed.
If there was no other reason than a philosophical one (the technical one is the GCC extensions) you could have built Linux with MSVC in 1997.
Reminder: the first two decades ended 11 (ELEVEN) years ago.
We are in the fourth decade right now.
> and why are you quoting RMS while talking about development of the linux kernel?
Because Stallman was in charge of GCC and the history of GCC and Linux are almost inextricable from 1991 onwards.
I can't cite sources, but AFAIK Linux uses GNU compiler extensions. The reason is not to lock other compilers out, it's just that some of those extensions are genuinely useful.
Of course.
They are genuinely useful to Linux.
Other did not use those extensions because they did not want to use GCC, for reasons beyond the technical merits, but because GCC is free software
If they could copy them, they would have done it.
It’s just that Linux used GCC extensions and no one was interested in doing the work necessary to have it compile on a non-GCC compiler.
that's total BS
are you telling me that Alan Cox had no involvement in the Free Software?
https://en.wikipedia.org/wiki/Alan_Cox_(computer_programmer)
> It’s just that Linux used GCC extensions and no one was interested in doing the work necessary to have it compile on a non-GCC compiler.
the GCC extensions were essential to enforce the GCC supremacy because no other non-free compiler could implement them
I don’t see how your statement contradicts or is even linked to mine.
The kernel community as a whole very much has little interest into the philosophical arguments surrounding open source. Apart from being convinced that sharing code is the best way to develop a kernel they have next to no active involvement in the whole charade.
See for exemple keeping GPL v2, not opposing TIVOisation, disapproving on technical merits but allowing proprietary drivers and binary blobs.
> the GCC extensions were essential to enforce the GCC supremacy because no other non-free compiler could implement them
Linux uses GCC extensions because they are handy and GCC was the compiler everyone used to compile C projects for a long time. It’s not intentionally done to promote GCC on ideological ground, something pretty much no one cares about in the kernel community.
because they didn't have to.
someone already established that it was the foundation, people like Linus Torvalds, Alan Cox, Maddog Hall and many (not too many, actually) others.
The "community" for the longest time has been a bunch of people
"is a small and well-defined group: Linus, Maddog Hall, Alan Cox, and somewhere between 6 and 12 others (varying at times)." (Steven Suson, 1999)
"Watch the linux-kernel mailing list. The "Inner Circle" becomes very obvious. People go in and out of the Circle, so a list probably isn't possible [...] I would say it includes may be 2 dozen people." (Eric Princen, 1999)
> It’s not intentionally done to promote GCC on ideological ground, something pretty much no one cares about in the kernel community.
again: you're talking at the present, I am talking about the first two decades
you may have forgotten about it, I did not.
Who said anything about zealots?
please, stop putting words in someone else's mouth.
Linus was a big supporter of free software and the fact that the Linux kernel was free software is what compelled many developers to donate their work for free
They didn't do it to improve NT Kernel or Solaris kernel or... you know it.
---
Software is like sex: it's better when it's free.
-- Linus Torvalds
But Linux didn’t go out of its way to be incompatible with ZFS out of ideology. Sun intentionally picked a license which would not be compatible. That’s why ZFS lives outside of the main tree.
The ideology that’s driving the Linux kernel development: “shared source code leads to better code, user land should never be broken” are very different from the one that led to GCC gimping itself. Generally speaking the kernel community is very technically oriented and doesn’t go out of its way to prevent things for social reasons.
EXPORT_GPL_ONLY
The kernel does care about enforcing the GPL explicitly on some of the interface it presents to modules to ensure the openness of the code which is indeed a form of statement in favour of open code.
I don’t think the situation was the same regarding compiling only with GCC - after all clang is free software - and I think the heart of my argument still hold: the kernel community makes decisions mostly for reason related to the kernel - even there they just want to force code to be mainlined - rather than for the movement like the FSS.
I think you mean it just isn't finished yet, but it's almost there.
> It’s awesome that we can already compile Rust for Linux with very few hacks: we’re very close to being able to compile it on the master branch.
https://blog.antoyo.xyz/rustc_codegen_gcc-progress-report-16
They either don't compile the rust code, which is fine, or they wait for gcc support.
> What about the billions poured into LLVM that made writing an optimizing Rust compiler possible in reasonable time, that is still written in C++?
I'm not sure what you're asking here.
> Lastly, I hope to be wrong on this, but watching at Google history being backed by them is, unfortunately, a course,
Also unclear on what this statement is about.
(OK, a bit sarky, but still, Rust isn't a magic bullet...)
Of course it won't fix all those half-assed logical bugs, but it will put an end to memory and thread related bugs, of which there are many.
And the reality is that there are no absolute guarantees. Ever. The "Rust is safe" is not some kind of absolute guarantee of code safety. Never has been. Anybody who believes that should probably re-take their kindergarten year, and stop believing in the Easter bunny and Santa Claus.
Torvalds is merely stating the obvious and being a obnoxious dick-head at the same time.
So even the "rewrite it in Rust, badly" has a decent ring to it tbh (although I'm not arguing there should be a rush to so).
> “After having thoroughly debugged the orbital/orblogin memory corruption bug with little success, I decided to go as far as phase out the old paging code (ActivePageTable/InactivePageTable/Mapper etc.) in favor of RMM (Redox Memory Manager). Surprisingly, this fixed the bug entirely in the process, and it turns out the issue was simply that parent page tables were not properly unmapped (causing use-after-free), most likely due to the coexistence of RMM and the old paging code, which did not agree on how the number of page table entries were counted.”
(https://www.redox-os.org/news/drivers-and-kernel-7/)
This project surely uses rust more thoroughly and idiomatically than the Linux kernel ever will. And yet here we are with memory corruption and use after free bugs. And the text indicates the bug was so hard to track down that they basically gave up and just replaced the old code.
Rust may prove to be beneficial to Linux, but there is too much over-promising hype at this point.
Also, you need unsafe() blocks once in a while to interoperate with the extremely low-level assembler parts.
So I'm not in the least surprised there are still memory corruption errors in Redox.
Mind you the bug occurred in the memory paging code and was eventually fixed.
Whether it's an unsafe() block, or that in a kernel a logic-bug can also be a memory-corrupting bug (and apparently a use-after-free bug), the fact remains: there's no magic bullet here. Just hopefully an overall improvement. We'll see.
https://www.destroyallsoftware.com/talks/the-birth-and-death...
Jokes aside, I wouldn't be surprised to see WASM in the kernel one day, perhaps to replace eBPF.
This may not be too crazy of an idea, it would enable fine-grained secure sandboxing like what Firefox is doing.
Securing Firefox with WebAssembly - https://hacks.mozilla.org/2020/02/securing-firefox-with-weba...
Practical third-party library sandboxing with RLBox - https://rlbox.dev/
The whole point of eBPF is that it's a limited subset of an execution environment that not only provides a memory safe environment, but just as crucially all eBPF programs are guaranteed to terminate.
It's not a general purpose computation environment in a strict sense.
You need to realize that
(a) reality trumps fantasy
(b) kernel needs trump any Rust needs
And the reality is that there are no absolute guarantees. Ever. The "Rust is safe" is not some kind of absolute guarantee of code safety. Never has been. Anybody who believes that should probably re-take their kindergarten year, and stop believing in the Easter bunny and Santa Claus.
If you cannot get over the fact that the kernel may have other requirements that trump any language standards, we really can't work together.
Exactly. Some people act like we don't have the whole branch of "formal proofs" in CS. Memory safety is just once aspect of program safety.
Like, IMO, programs written in Coq, F* or even C programs verified by Frama-C are much more "safe" than Rust programs that advertise their "safety" on the mere fact that they are written in Rust.
Also the current head of the standards comittee is Herb Sutter.
We absolutely need something like Rust to cover our backs!
Mind you that Firefox, for example, has been around for 17 years. Every month I get security updates fixing multiple memory holes in the code each of which could potentially allow anyone to take over my machine. Seventeen years!! They must have fixed hundreds if not thousands of such bugs by now, and every month more are repaired by updates.
We'd have none of those had it been written in Rust.
Also mind the time saved by developers not having to hunt for such (hard to track down) bugs!
Except for memory issues in JIT-generated code from Javascript? Not all browser vulnerabilities are caused by memory safety issues in the code generated by the Javascript JIT, but several of them are (particularly "type confusion" vulnerabilities). Rust might help avoid vulnerabilities in the code generator, but not in the generated code.
The factor that actually generates the bugs is the tired person writing code they will never read again or use. The borrow checker won't help with that.
I write code with different types of bugs in C, Go, Rust, Javascript, and Verilog, for example. I rarely write code with no bugs, especially when I have no real connection to the code.
If you think there's the same amount of severe exploit from software written in Go/Java/Javascript and software written in C/C++, you are just factually wrong.
70% of exploitable bugs from large company products (M$, Apple, Google) ARE memory safety bugs.
One big reason why we have so many CVEs for memory safety bugs is that they are very easy to find with analyzers and easy to programmatically test. We currently live in a world where a lot of deployed code has not had the benefit of those analyses, but the attackers do. Hence the huge number of CVEs. Rust closes that asymmetry, which is significant.
It does not mean that we won't move on to a new class of exploitable bugs that show up due to a new class of analyzers.
Also, exploitation means a lot of things to a lot of people. The fact that it's really easy to crash a Rust program could also be considered an exploit (as a number of CVEs do).
Just using Rust does not save you from exploits. Using Rust well makes it easier to be safer.
1. You acknowledge others' reports that 70% of exploitable bugs are rooted in memory safety problems.
2. You acknowledge that Rust helps reduce the number of memory safety bugs.
3. You resist any conclusion that Rust therefore reduces the number of exploitable bugs.
4. The stated reason for such resistance is "but but but they might exist and you might not know about them and something something something about how analyzers have gotten better."
1+2 alone seem like a pretty clear open & shut case to me. Your (4) looks like grasping at straw to me. You complain about "without evidence," but 1+2 looks like pretty compelling evidence to me.
One also has to wonder what kind of evidence would meet your standard. What evidence would convince you? And is that evidence even obtainable?
The evidence that would convince me is ~10 years of a notable reduction in both total CVEs and total monetary value of exploits of that software (or a comparable piece of software). There is almost nothing that would convince me today that Rust is inherently more secure than C or C++ (particularly modern C++), except in the small class of applications that rely solely on memory safety for security and cannot use a garbage collector.
By the way, I do have an application running in production today that fits that requirement, and I wrote it in Rust.
> I am also asserting that the Rust borrow checker likely won't help with those latent bugs.
Kind of a silly assertion, no? Particularly given you've failed to characterize these "latent bugs" other than the fact that they aren't related to memory safety. You've also failed to provide any compelling commentary regarding whether these "latent bugs" are present in C or C++ programs. If you believe they are, then Rust still makes the situation better by reducing the number of exploitable bugs by reducing the number of memory safety bugs.
Methinks you are suffering to an appeal to ridicule. Your comments are written as if someone is saying "Rust will eliminate exploitable bugs." Nobody has said that in this thread. More to the point, I'd be willing to wager that nobody with any modicum of credibility has ever said that.
Onto latent bugs: part of the idea of latent bugs is that they are latent, as in we don't know what they are yet. I am just pretty darn sure that we haven't found all the bugs in computer software, or even all the possible types of bugs. Spectre didn't show up until 2018, and it was a totally new type of bug.
I am pretty convinced, after working on huge-scale systems that get fuzzed really well by their clients, that we have found less than half of the exploitable bugs in Linux (and other similar systems). I'm not suggesting that Rust will hurt here, just that there's no evidence that it will. They will probably show up in C and C++ code too, but you never know.
By the way, one class of vulnerability that Rust code tends to introduce when compared to C++ code is supply chain attacks. Cargo/NPM/Pip are attack surfaces. So far, the worst version of this that I have heard about in Cargo was that several packages started printing some harmless politically-motivated text on startup.
Also, let's get this straight: I am saying that the claim "Rust reduces exploitable bugs" has not been proven yet. Let's stick to that, shall we?
I don't know how one "proves" that claim. But it definitely has enough supporting evidence to be a very reasonable belief. I certainly believe it.
I just find your entire approach here to be so strange. On the one hand, you're demanding concrete evidence, but on the other, you're waving your hands about latent bugs and quantifying the number of known bugs with zero supporting evidence.
I think this is one of those things that's common sense and you're overthinking it.
"Do static analyzers like cppcheck, Frama-C, Coverity or NASA's ikos catch more bugs (including memory bugs) than idiomatic Rust code?"
We have to consider the added complexity of the Rust language and the massive man-hours required to re-write old C code-bases. It's a trade-off.
Now, if we focus on functional properties (proving that the code is correct), one terribly hard problem when dealing with real world programs is handling the shape of the memory. That can make some proofs awful and really hard to complete. In a language like Rust, you can get a lot of information about the shape of the memory and the memory separation for free thanks to the type system. That would make proofs really easier (this is for example what is done, but with far less precision than what the Rust type system could provide, in the different memory models configuration in Frama-C/WP and it can already dramatically improve proof performance).
It probably does not justify rewrite everything, nor changing verification tools chains that are in use in some critical domains. However, I would definitely love working on and working with a Frama-Rust tool ;)
Several large surveys have found that approximately 70% of bugs in C and C++ projects are memory safety issues.
For more complex things, arguing with the borrow checker has often led me to weird solutions that seem to work, but are unlikely to be 100% bug-free.
I would also like to point out that the surveys are looking at known bugs, and memory safety bugs are now very easy to find, thanks to the large number of memory safety analyzers that exist.
The analyzers exist because memory safety problems are significant, but the fact that they are 70% of known bugs does not imply that Rust code will have 30% of the bugs that you would find in a C equivalent. In my experience, if the C equivalent is short, Rust code usually has 0% of the bugs! If the C equivalent is long and complex, it could be a lot higher.
So you're telling me you don't unit-test your code? Big red flag there!
Rust is one of the few programming languages where unit testing is an integral part of the toolset.
What would you like elaboration on? The relationship seems pretty clear to me?
When you hit a limitation you really need to implement in Rust, they say "Hey, there's unsafe{}, use that". Also, they advocate that unsafe{} is equal to C/C++ in programming freedom, which is again not.
When they're reminded that reality is not like that, they get upset and defensive. This comment is a nice flag to remind this reality.
I congratulate Rust for being what it is, but it's not a silver bullet and it's not the next C or C++. It's just a very nice programming language for various applications.
Being all shiny-eyed doesn't work in CS or programming in general, and also hardware doesn't work like that (a deterministic, perfectly good behaving, spec-obeying magic box with some included smoke for higher performance).
Can you point to (high-profile) evangelists that actually push this view? I think the people you refer to are a loud minority of developers who possibly don't even write professionally in Rust. I've never read or heard this position from experts.
Maybe the messaging should be clear, and hype should be better controlled from top to bottom, because I never talked with anyone who doesn't casts stones to C & C++ and badmouths it while saying that Rust is the one, and only, and the last savior we need.
BTW, I'm not against Rust, but I don't use LLVM. I'm waiting gccrs to dive into the language.
Second aspect is I don't like the behavior of LLVM ecosystem which is trying to subtly EEE gcc toolchain.
Lastly, I don't desire to be able to build a project I developed, or cloned to be only can be built with "clang/llvm/20201024+somecompanyClosedGitBuild20210514+bp1" which is available in binary form for a single distribution for a specific architecture. I have experienced SDKs and other software like that, and I don't want to go through again, or put anyone through that hoops.
As opposed to GCC, which not-very-subtlety added extensions onto C which hurt portability?
GCC really shot themselves in the foot by making their architecture so difficult to use as a library, even from GPL code.
So here is one, Ada was already that saviour in 1983, and in every use case where using a GC is an option, there are already plenty of options besides Rust.
Also until Rust compilers get fully botstraped, knowing C++ is a requirement to contribute to the compiler's backend.
There are plenty of options besides Rust, but none that have & and &mut which eliminates many classes of bugs in both multithreaded and single-threaded code.
https://www.adaic.org/resources/add_content/docs/95style/htm...
You are not supposed to call Ada.Unchecked_Deallocation() directly, rather from Ada's version of "Drop".
Then SPARK allows for formal proofs that one actually does the right thing, integrated into the language as of Ada 2012.
The cherry on top of the cake is that the language is also getting affine types on top of the already battle tested features in 40 years of production deployment into high integrity systems.
Yes, SPARK exists, and is very cool. I think there's work on a similar system for rust (or ferrocene, anyway, which is being co-developed by AdaCore: https://blog.adacore.com/adacore-and-ferrous-systems-joining...). It also seems that SPARK now has memory ownership and aliasing support, but only since ~2019 (https://blog.adacore.com/using-pointers-in-spark), and with clear inspiration from rust. It seems to me that this addition to SPARK was felt necessary by the AdaCore people, meaning that rust had something Ada did not support (or did not support well).
Rust isn't a silver bullet but it does shift the balance materially.
I do definitely cast stones on C/++, especially considering that a C project I've rewritten in Rust uncovered several memory safety issues from the very first run. Another C project I've approached also showed C-specific issues from the very beginning.
On the other hand, I don't define Rust and the savior and so on; to be specific, memory safety is a very serious issue, but not the only one, and memory safe languages solve a very serious issue, but not all the issues. Additionally, I think Rust is a niche language; other memory safe languages may be more productive for the majority of the use cases ("webcrap", like a famous software engineer described :)).
Now you know one!
Where are these people? I'm not doubting that they exist, but I've never come across one and I've been in the rust community for several years now.
All memory-safe languages in existence have safe abstractions built around unsafe code. For example, most JavaScript runtimes are written in C++. The point of unsafe Rust is to be able to use the same language to write both safe and unsafe code.
On the flip side, almost all C and C++ is unsafe code. The point of unsafe Rust is to be able to reserve brainpower for small subsets of code while having a relaxed posture towards most of it.
I'm interested. How does rust unsafe give you less freedom than let's say c or cpp?
There are often ways to do whatever you want, but it's not as simple as "I have a raw pointer now let's goooooo."
I'd describe it as different, not "less", but I wasn't the one who said it originally.
It’s not that Linus is against Rust at all, just clearing up some disagreements.
(don't come at me its just a joke)
Unless that 0-day comes from some other software, it seems unlikely that we'll get such a worldwide supply chain issue.
Being programmed in C or Rust is a non discussion. Using both in the same kernel source complicates code unnecessarily.
"Complicates code unnecessarily" can be met with a counter that it's necessary for the memory safety of the future millions of devices out there.
If Microsoft and the Linux Kernel release security updates periodically, and people's research says that around 70% of those updates are addressing memory safety issues, "unnecessarily" starts to sound more like a necessity.
We have to get to a point where resistance for the sake of resistance becomes unproductive.
C is a poor choice for large software projects, because of its fragility (high-level assembly, remember).
The real problem lies in the fact that some obstinate developers refuse to trade even one iota of performance for security. Without that we wouldn't have the mess we're in today! It's relatively easy to develop a programming language that's memory and thread safe, but at the cost of some performance.
That's why Rust was developed: to get the same performance as C without the drawbacks. It does this by simply disallowing you to write dangerous code. Underneath it's simply C code, if you will.