Write your first Linux kernel module | Hacker News Reader

Write your first Linux kernel module

linuxvoice.com

287 pointsbenev44 comments

This is so great!

Writing toy kernel modules on your own is a super fun way to get into kernel programming. There's no danger of anything going wrong, especially if you develop inside a VM.

My partner wrote a kernel module for a talk I gave that rickrolls your file system -- it overrides the 'open' system call so that any time you open a '.mp3' file, it plays Rick Astley instead. (this requires you to have Rick Astley already on your computer =))

The source code for that (and a few other toy kernel modules) is at https://github.com/jvns/kernel-module-fun

The rickroll module (by Kamal Marhubi) is the most well commented and fun: https://github.com/jvns/kernel-module-fun/blob/master/rickro...

I wrote a blog post with more about this kernel-programming-for-fun philosophy: http://jvns.ca/blog/2014/01/04/4-paths-to-being-a-kernel-hac...

Of course, writing code for the mainline kernel is a responsible task, but writing kernel code on your own machine that nobody will ever run in production is a great way to learn, and I think it's made me a better programmer.

A few concrete ways I think it's made me a better programmer:

- I now understand the interface between kernel space and userspace much better (I use strace all the time to debug my regular userspace code!)

- realizing that the Linux kernel is just written in C and it's something I can read (and understand! and change!) was a big deal for me

- understanding the basics of the underlying operating system helps me reason about my programs better

- I now feel like if I ran into an actual kernel bug or performance problem, it's something I could conceivably attempt to fix and understand, instead of hiding under the bed and saying OH NO THE KERNEL IT IS TOO HARD

This is my first kernel module, which 'fakes' your system uptime:

http://www.anfractuosity.com/projects/uptime/

Personally I think people should be encouraged to hack around with the kernel, rather than be afraid of it, how else can we learn?

This is a good starting point for anyone interested in the Eudyptula Challenge [1].

[1] http://eudyptula-challenge.org/

The idea for this came to us after a long night of drinking in which it was determined that if the Linux kernel was to survive, it would need new programmers to fix all of the bugs that were recently added after a long night of drinking.

Sound thinking.

Do most kernel programmers start out working on their regular daily-use machine? My only experience is with an embedded linux environment, and I would probably recommend starting with something like a RPi so that your environment is easy to restore if you mess something up.

I also recommend "Linux Device Drivers" from O'Reilly. http://lwn.net/Kernel/LDD3/

> I also recommend "Linux Device Drivers" from O'Reilly. http://lwn.net/Kernel/LDD3/

That book is a little out of date nowdays.

They're working on LDD4: http://www.amazon.com/Linux-Device-Drivers-Jessica-McKellar/...

but oddly, it's based on kernel version 3.2, which by the time the book will be released is also going to be out of date.

Whenever I do kernel stuff, it happens in a VM, limited to a subset of the cores available on the machine. That way when I inevitably bork something, I can just forcibly stop it.

Another trick I've used is making as much of my core logic as possible happen in functions that are called from the kernel hooks, rather than in the kernel hooks. Then you can test a ton of it in userland to handle basic logic and flow control bugs, finally porting the code to the module. To help with this, it sometimes is actually faster to copy some of the kernel routines you use to a different .c file and change things like kprintf and kmalloc for the userland versions. It isn't going to be a perfect port, but it helps get some of the complex stuff out of the way where you have your familiar userland tools.

All that said - I've only dabbled in the kernel a little, there are probably better ways that someone can chime in on.

Idea: sysctl/ioctl rest interface so one can drive the new kernel module from the host vm.

Keeping the complex stuff out of the kernel is good advice. (and not just during development)

True! Unfortunately it's sometimes unavoidable. The specific case I was remembering involved writing a qdisc... it was actually more complex to move that out of the kernel via nf_queue than to just do it in-kernel. (At least some parts of it.)

I agree, that's a great book to start with. When I started out programming the Linux kernel, I thought I had to understand everything and I hated the book for not going into detail on how things worked. But really, I just had to start changing code and running it, and I became a Linux kernel programmer. Now I see that I just needed to know the basic roadmap, and the rest comes from reading code and experimentation. It doesn't matter much if the book is a little outdated; the core concepts stay in place for a while. Another book I'd recommend starting with is The Linux Programming Interface. You can write programs for a while without getting to the kernel/userspace boundary, but this is essential to understand if you're going to write user-facing code (e.g., a driver).

It's just like the article said--just a big, annoying to debug C program. If you understand pointers, you can learn how to program the Linux kernel. Programming in the Linux kernel has been a really fun learning experience for me. It's nice, if intimidating, how you can just do everything--there are no hard abstraction boundaries, you can read all the source and call into any part of the kernel without going through hoops.

I've developed the Linux kernel with two machines, a development machine and a headless machine that I'm ssh'ing into (actually, a bunch of those--makes testing go faster). VMs aren't really an option for device drivers, but if you run your code on your development machine, you waste a lot of time.

If you're doing something with a device driver, there's not much alternative, since some hardware tends to get into a state where only a hard reset will fix it.

Save and sync often, and have only the bare-minimum of processes running.

I might be missing something, but the final version still looks susceptible to deadlock to me.

Suppose the buffer is empty. Process A, which is reading, gets through the mutex_unlock() call and into wait_event_interruptible(), which seems to first internally check the condition. It is then switched out before it internally waits.

Process B then runs all the way through, acquiring the mutex, doing its work, and runs wake_up_interruptible(), which won't do anything, because process A hasn't started waiting. Process B then releases the mutex and gets switched out.

Process A then gets switched back in, and then immediately starts waiting, with no more events to wake it up. Why won't it just wait forever (deadlock)?

Am I missing something? Do all processes occasionally get woken up without an explicit wake up?

VirtualBox uses a kernel module, and that is annoying because the linux kernel API changes with every update, which means its kernel module needs to be recompiled. Since I update my Linux quite often, that basically means:

Everytime some friends call me to play a certain game online with each other, they need to be waiting a bit longer for me because I have yet again to recompile VirtualBox's kernel modules.

Is there a way to convert something that requires kernel modules, to run fully in userspace?

I'd be happy to lose some efficiency for this. Just a 100% reliable game to run my game would be perfect.

Are there things you can do only in kernel modules and not in userspace, and does VB need this? In theory userspace can do everything you need, right? Graphics, networking, audio, ...

Virtualbox is an out-of-tree module and hence needs to be compiled. The base package is GPLv2 and is compatible with the Linux kernel license, but I don't know if Virtualbox developers ever tried to "upstream" their module. This is probably the easiest way to solve your problem. Most Virtualization Software these days use virtualization hardware extensions (Intel VMX/AMD SVM) and so the "hypervisor code" needs to be able to execute privileged instructions, eg - you can't run VMON from ring 3/userspace.

Apparently the virtualbox code is terrible, hence the lack of upstreaming...

I'd just update the kernel less if i were you. There's not really much of a point updating the kernel all the time for normal use is there?

Well, it's Archlinux, and the most basic usage pattern is to use "pacman -Syu", which updates everything (useful for updating your programs), including the kernel.

There is a kernel-lts packages that doesn't update as often, so you won't need to recompile every time.

In Arch, failing to regularly update breaks your system. Also, regularly updating your system will definitely break it at some point (bleeding-edge dontcha know). The official position is that it's your own damn fault for upgrading or not upgrading, as the case may be.

Arch is broken by design.

DKMS might help if you have access to the source.

https://en.m.wikipedia.org/wiki/Dynamic_Kernel_Module_Suppor...

I had great fun writing a silly toy kernel module a few weeks back, ( https://github.com/benjojo/dev_markov ) Would totally recommend anyone curious giving it a go.

Cool, definitely relevant to my interests. I've done various ARM/orion5x platform porting related stuff before but never a module...

Really great article.

Maybe it's worth emphasizing that writing kernel code is a very responsible task, not to be taken lightly. Hackers coming from a userspace background tend to think it's just "userspace with a few quirks" which is grossly wrong: your code has to be completely free of bugs, as a dangling pointer, for instance, won't just send you a friendly SIGSEGV and terminate your ass, it will crash the whole system, leading to consequences for everyone who relies on its stability. Just imagine introducing a bug which crashes a Linux-based life support system. You could be responsible for endangering human lives. (I ageee that Linux isn't a sane choice here, but just for the sake of the argument.)

Also, standards to get your pull requests merged into mainline Linux are insanely high, but even given that, you can never rely solely on maintainers' code review to catch your mistakes. Take kernel hacking seriously.

That's putting it a bit harsh. Yes, if you plan on writing kernel code for your fleet of production systems, or for submission to mainline, you need to be very careful.

That said, starting out, you work on a development system, you write a bunch of code, it'll have a lot of bugs, and you'll learn from them and from the process of fixing them.

Saying "You could be responsible for endangering human lives" is quite overdramatic. If people only pushed code to the kernel that they knew 100% was "completely free of bugs", we'd not have a Linux kernel at all. I've had to roll out enough new kernels due to security issues just this year to know that.

Yeah, on second thought, it probably sounds over-dramatic and all, but it would probably impose some valuable discipline for people who want to hack at the Linux kernel. (I've been programming for quite some time, but have never thought I am ready for going beyond userspace. Maybe I have it a little wrong, after all.)

I know you meant well, but to give such a stern warning and then admit that you haven't done any kernel-related work at all is frustrating. I've just been getting involved in kernel development for the past month, and it's pretty clear that the go-to taskss for novices is primarily writing device drivers or fixing bugs. I'm not sure what fraction of kernel developers get to work on the kernel core (CPU scheduling, boot/initialization, file systems, memory management), but I'm going to guess it's relatively small, or at least isolated compared to those working on drivers and other modules.

Further, if the maintainers were so cautious as to want to limit contributions to the kernel, you wouldn't see initiatives like the aforementioned (and awesome) Eudyptula Challenge, or Greg KH's "Linux Kernel In a Nutshell", which he chose to give away for free in order to garner more interest in kernel dev. The state of the project right now is clearly weighted towards getting new involvement as opposed to discouraging contribution because of extremely rare and unlikely cases like the one you mentioned.

I don't think you have to do kernel work to warn people they should be cautious when submitting to mainline. As a matter of fact, I've read the book you mentioned, alongside some others such as Understanding the Linux Kernel by Bovet and Cesati, and I've brushed against low-level kernel features such as memory-mapped I/O or epoll, so I'm at least aware of what might go wrong if you decide to write a kernel module just for fun and then issue a pull request against the original source tree.

I don't think contribution should to be discouraged, but I do think that people who have no sense of responsibility should be kept away from the kernel source. Kernel development is not supposed to be for everyone, it's hard. Great expertise and technical mastery is needed to write good, quality and responsible code that is suitable for inclusion into a kernel that today powers so many different devices.

I will admit, again, that my warning was probably too harsh, but I still stand firmly by the viewpoint that "casual" or careless kernel contribution should be a no-no. It's one thing to poke around on your dev system to find out how stuff works, but entirely another to write an official patch to a system that millions of people around the world rely on.

EDIT: And when I say that "kernel development is hard", I mean that it's harder in the general case than writing most userspace code. It requires you to think of stuff you don't usually worry about (think of the all the race conditions that might occur) and let go of all the abstractions provided to you by the kernel, it's harder to debug than userspace programs, and you have to support a wide variety of hardware platforms, each with its own quirks. In fact, I haven't done any kernel development just because I was intimidated by how hard it is. I've tried writing a kernel from scratch (OSDev wiki was a great help) but it soon came to be a buggy and unmaintainable set of kludges that was barely doing any real work.

EDIT #2: Why I can't reply to replies to this comment? Is there a conversation depth limit on HN?

"Kernel development is not supposed to be for everyone, it's hard."

Citation?

Given you've admitted to not working on kernel code, this seems like something you've determined out of reverence for the importance of the kernel, not something grounded in fact.

It's worth noting that the entire Linux project spawned from some "casual" development work by somebody doing it as a hobbyist project.

Your last point is just, ugh, oh so true. You don't get to be an expert in any field without diving head first into them at some point as an inquisitive amateur. Having a curious, naive spirit who isn't afraid to do things "the wrong way" and break a few things along the way is something we should be encouraging in our fellow hackers!

The OP's misaligned sense of responsibility honestly terrifies the child in me. I would not be where I am today were it not for the countless circuit boards I went through (nice way of saying completely destroyed) when I was 12.

True. If you follow LKML, you will notice that the majority of changes are to the device drivers code. And it makes sense, after all, you want the kernel to run on a wide variety of hardware. So, it's not just what fraction of developers work on the core but rather, what the fraction of changes belong to the core kernel - which is relatively small compared to the device drivers code.

Another good way of getting into kernel hacking is not just device driver books but following mailing lists you are interested in. netdev, pci-devel, kvm are some of the interesting ones. Also, checkout GregKH's driver project - these are drivers that are not yet fit of inclusion and hence reside in a "staging" area - a great way to get involved in kernel development ( http://linuxdriverproject.org/mediawiki/index.php/Main_Page). Note that some of the lists can have really high traffic but eventually you will get the hang of picking out the useful stuff. https://lwn.net/Kernel/Patches is a great resource for mention-worthy kernel patches without subscribing to individual lists.

I also agree with you that maintainers will _not_ be cautious and prevent newcomers. That's a bunch of bull. If at all, maintainers will be welcoming to changes from first timers and will thank you for your contributions if your changes do get in. Maintainers or other developers usually are frustrated with other style issues though - for example, lines greater than 80 chars, using a mail client that mangles your patch and so on. But keep in mind that the frustration is on the patch you posted, definitely not on you on a personal level :)

Hm. I'm a Linux kernel subsystem maintainer, and I think this stereotype ("only extremely accomplished programmers should work on the kernel") probably hurts more than it helps.

I'm not sure I've ever met someone who wanted to contribute to mainline and didn't understand the responsibility involved. What I think we see more of is excellent programmers who think they aren't good enough for the kernel, with the result that the average age for core kernel developers appears to be increasing all the time. It's not a great situation.

I'm not saying my opinion is representative of all kernel maintainers, but I'm personally in favor of losing the "you must be this tall to work on the kernel" rhetoric.

It's probably more to do with personality type than intelligence or anything else. It takes an extremely thick skin to handle the regular patch review slating dished out by most of the core team.. I simply don't want to deal with that shit for a hobby, and it'd take an impressive salary to make me deal with it professionally.

There's being opinionated, then there's being an ass, and I think there's far too much of the latter on the kernel lists.

If you look at the modern nerd, compared to the generation that birthed the kernel, they're more socially aware, fashionable animals that are more interested on working on sexy, visible things. Why spend your day dealing with bad attitudes and timing bugs when you can pick up $200k+ writing Ruby apps that enable sharing 5 second video clips?

I doubt the kernel will ever be starved for new talent, just that the ratio of people willing to work on it or deal with the social norms it entails is shrinking all the time.

I think a better quote would be:

"There's being opinionated, there's being an ass, then there's Linus Torvalds."

Don't worry though, he's not writing a three-paragraph rant calling you stupid, he's calling your idea stupid! Why do people take things so personally!?

...big ol' "/s"

Maybe the lkml is not always so bad, but I'd rather not deal with it and so far I have been content playing with my own projects. I feel kind of silly complaining about it when so many people manage it just fine, but then I see another explosion of aggression and drama and think "how are people okay with this?" I can't imagine having to work in an environment like that.

And then there's Theo de Raadt :D

>You could be responsible for endangering human lives.

I'm pretty sure what ever devices people depend on to keep them alive would be running a modified version of the Linux kernel with anything unnecessary stripped out and probably a lot of custom stuff.

They wouldn't just download it from kernel.org and throw it on there. I would hope not at least.

I wholeheartly agree with ❝devices people depend on to keep them alive would be running a -- version of the Linux kernel with anything unnecessary stripped out❞.

But the part with a ❝modified version (...) and probably a lot of custom stuff❞ made be shudder.

The Linux kernel is a very well maintained piece of software, and whenever, in the embedded world, I've seen modification or larger forks of the stock/mainline kernel, they always were of worse quality and had more bugs than what I (or everyone else) has gotten used to when running mainline code (as is customary on your normal x86 PCs).

That's because code-review actually works in Linux, and while it makes it sometimes very hard to get new features in, the ones that make it are usually well vettet.

I agree. But I think kernel contributors should do their absolute best to ensure the code is as bug-free as possible. There are irresponsible sysadmins who'd use a Linux kernel where there is no place for a Linux kernel to be, and just because of that alone it would be a good idea to be paranoid when pushing to Linux mainline.

Not everyone writes code for a critical Linux-based life support system. Very often it's just people who want to have a 7-segment LED display show the number of unread emails, or get a soundcard running, something along these lines.

And for them it's completely ok that there's a race-condition that might crash their machine if the kernel-module is unloaded just at the instant where the USB plug is removed -- that's something to learn, once the general grasp of how kernel-code with it's myriad of callbacks and hierarchies of subsystems has been reached.

Most crashes also will be catched by the Kernel and will leave you with nothing more than a

    [42568.139488] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
    [42568.139805] IP: [<ffffffffa003e008>] init_module+0x8/0x20 [my_dummy_module]
    [42568.140308] PGD 4450f067 PUD 4450e067 PMD 0
    [42568.140882] Oops: 0002 [#1] PREEMPT SMP
    (...)

in your dmesg, and maybe a dead "insmod" process in your process table. Make a note of the call trace, sync, reboot (maybe hard, using the magic sysrq key) and you are off for the next round of debugging. Try it, it's fun. Also there's kgdb which you can use when your bugs aren't that obvious.

I never had serious data loss (maybe the freshly compiled .o files will have 0-size, if I was in too much of a hurry to insmod ./my_module.ko. And even if, it will teach you to have an easily restorable development machine.

(I triggered the oops with...)

    int
    init_module(void)
    {
            int *ptr = 0x0000000000000004ULL;
            *ptr = 17;
            return 0;
    }

> Very often it's just people who want to have a 7-segment LED display show the number of unread emails, or get a soundcard running, something along these lines.

7-seg LEDs are best controlled by a microcontroller attached via USB or a USB-RS232 interface, as next to no one ships a parallel printer port. Soundcards these days usually all already have kernel drivers, I haven't heard of any modern chip without.

And USB stuff is best developed in userspace with libusb, it's way easier...

> it will crash the whole system

As anyone who's programmed in "completely open" environments like C64, ZX, DOS, etc. will be accustomed to, reaching for the reset button becomes almost second-nature after the first few times you mess up, and then you start developing a more thoughtful approach than just "compile and run, see if it works" when you realise that waiting for the reboot takes a significant amount of time. Maybe this is some sort of counterargument to the trend towards "early and often" testing cycles...

But I really think everyone should have attempted writing code for an environment with no memory protection and where bugs mean a reboot. It probably makes you a better programmer overall.

I agree with you completely. We shouldn't live in the past and fail to embrace the present, but I think everyone would benefit from knowing what the past was like. Maybe it will make you develop quite a few healthy habits, after all :)

Reply on news.ycombinator.com