Hacker News Reader

I think this is a typical reaction from someone with no clue what is going on inside.

> Why are fundamental things like resource allocation at least not standardised in the kernel?

I'm an outsider myself, so take this with a grain of salt. But having implemented a few datastructures in C, this gave me a little chuckle. You can't "standardize" resource allocation. To be efficient and organized you need to tailor allocation to the problem at hand. I bet there are at least half a dozen common strategies for allocation in the kernel. (And most individual methods are of course "standardized". Do you suggest kernel developers are too stupid to see obvious improvements?)

What you can, and probably should, do is have all elements of one type allocated/managed through the same mechanism.

I wasn't suggesting there is necessarily one single mechanism for performing memory allocation. I'm aware of arena allocators and that sort of stuff, having used them in the past and having been an avid reader of GC/compiler literature in my youth.

What seems to be the issue here is that the ownership of memory is not clear. The author seems to think they own it (it's not clear from the blog post if they allocated the memory but I assumed they did) while some other part of the kernel is freeing the memory. There are ways to track ownership of memory so this cannot occur. Rust does this. It's independent of how memory is allocated. A weaker form is to use some convention, which doesn't seem to be the case in the example described.

The other issue I have with your response is what I see as a typical knee-jerk reaction whenever the kernel is criticised: your calling me clueless and saying "Do you suggest kernel developers are too stupid to see obvious improvements?" I.e. to suggest that any criticism of the kernel comes from a place of complete ignorance and kernel development could not possibly be improved.

Kernel developers are not gods. They are not perfect. They are not above question. What I have seen is a lot of arrogance amongst kernel developers, the few times I've dipped into kernel development, that makes me think they probably are missing some obvious improvements.

This paper: http://www.ece.ubc.ca/~sasha/papers/eurosys16-final29.pdf is both a good read and an example of missed improvements to the Linux kernel scheduler, and it does suggest (though doesn't conclusively demonstrate) that these improvements are missed due to the arrogance of the kernel developers.

Fair! Just wanted to point out they already "standardize" within reason, i.e. function abstraction per datatype and the relevant documentation.

These types of problems (getting ownership wrong) might be common, but whether they can be solved using a mechanism like Rust's remains to be shown. I doubt it, but I'm not intimate with Rust.

(Personally most Rust programs I've seen so far are off-putting at a syntactical and sometimes architectural level - the latter is of course not entirely the language's fault. Kernels will probably not be one of the first areas of Rust adoption).

A reasonable discussion on the Internet! I'm shocked (but thank-you) ;-)

I too find Rust a bit off-putting on the syntactical level, though architecturally what little Rust I've looked at does appeal to me, as I'm in the functional programming camp from which Rust draws inspiration. I'll hopefully get to do some real work with it soon, which will give me a better feel for it.

Agreed that the power of Rust's borrow checker remains to been shown on a large scale. I'm optimistic it will be, but time will tell.

> Kernels will probably not be one of the first areas of Rust adoption

You probably mean popular/used-in-production kernels, but there are OS's being written in Rust, e.g. Redox [1]

[1] https://github.com/redox-os/redox

I had heard of that project. I'm sure there are OS's in Haskell too, but yes, that was not what I meant.

> The author seems to think they own it

Yes, fundamentally this appears to be the issue, and I agree entirely that the concept of ownership is something you can get, albeit a not great version of it, in a standard way. C++'s unique and shared pointers, as an example, make ownership concepts much clearer. Why is the Linux kernel using raw pointers? Even just having a unique pointer would provide tons of help.

This is not an uncommon issue. The Linux kernel is plagued with use after free vulnerabilities and invalid dereferences etc. But there's very little upstream movement to solve these problems.

I think unique pointers are basically RAII for pointers, which does not help for lifetimes that aren't controlled by lexical scope. Shared pointers have reference counting overhead, which I think is used mainly for concurrent access in the kernel. Furthermore often kernel people want to "see" and control where stuff happens.

unique_ptr helps even outside lexical scope when transferring ownership through function arguments, it makes it much clearer that ownership is transferred as opposed to borrowed. It can also be controlled manually (not only by scope) by manually settings its contents to nullptr.

One issue with Rust (which I know you weren't actually suggesting for use in the kernel) is that it's difficult to implement recursive mutable data structures in a way that makes the borrow checker happy. Recursive mutable data structures are pretty common in the kernel, I should think.

I thought the idea with rust was that it was low-level enough that such problems could be expressed with zero overhead in an unsafe block, and then given a safe interface for users further up the chain?

There's nothing wrong with scary-potentially-misused-unsafe stuff as long as you can isolate it and therefore less likely misuse it unsafely...

C effectively is unsafe everywhere, and other languages that are safe are non-zero-overhead, or so far up the stack that they cannot be used in kernel/system level stuff.

Is that right?

Yes, your parent is saying that it would be nice if Rust's safe subset could understand this, so you wouldn't need to resort to unsafe. You can absolutely do this stuff with unsafe, it's just, well, unsafe.

How could a safe system actually do it? Without some kind of run time garbage collector? Doesn't this kind of problem really require run time (either manual or automatic) care of memory by definition?

You can do it inefficiently and kinda manually in rust via weak pointers.

Otherwise GC is basically the only way in rust. You can have tightly-scoped GC in rust that doesn't affect the runtime of everything else, though. No good such GCs exist right now.

There are some type system tricks in hypothetical type systems you can do to make it possible to deal with these without runtime tracking. You can use the type system to ensure Detach traits get implemented, where Detach is for breaking cycles in a pass before destruction.

To be clear, when stuff's implemented in a safe block, it's usually implemented correctly, ie safe to use. It's just the logic behind it is harder to track than the normal lifetimes stuff.

I'm betting you could do something with a dependent type system - from what I understand you can encode the state of data as a type... but I'm not really familiar enough to say for certain. Take this comment with a huge grain of salt.

If we knew the answer to that, then maybe safe Rust would be able to handle it :)

Or rather, knew of one without a GC.

"Learning Rust With Entirely Too Many Linked Lists" is a great example of the complications you run into writing such a data structure: http://cglab.ca/~abeinges/blah/too-many-lists/book/

This is great, thanks for linking to it. But yeah, Rust may be the first programming language where you have to read an entire book before implementing a linked list! (I do like Rust a lot though.)

> Why are fundamental things like resource allocation at least not standardised in the kernel?

They are. The slab allocator mentioned in the article is one part of that. The problem here was not that allocation is ad hoc, but that the author was unaware of the conventions.

It would, of course, be nice to have some static analysis to help out people in that situation.

I think it's poor methodology combined with a "dangerous" language: lack of unit-testing/TDD, low-level verbosity, lack of formal behavior prooofs, kitchen-sink features lumped into the kernel, constantly reinventing the wheel and lack of minimal package code re-use.

Don't get me wrong, I learned all the sharp edges by trial-and-error in the C89 days with Pascal and embedded assembly, but I think we've got to have systems languages which provide higher-level abstraction for bare-metal kernels. It's just that even codebases like OpenBSD don't go far enough like seL4 to verify their security model beyond reasonable doubt with formal methods.

> The worst thing about the current situation is the programmer blames themself ... rather than wondering why this problem is even possible in the first place.

Absolutely agree. I think environment is partly to blame. Devs fight fires rather than prevent them because it makes their efforts more visible.

It's the reason you can demand metrics to justify everything - the only metric that justifies defensive programming are examples of bugs/failure, and it would be better to just avoid them in the first place.