Hacker News Reader

Comment by EdSchouten | Hacker News Reader

Notice how “unit testing coverage” is not on this map.

Unit testing is generally not useful for bare-metal code, since it depends more on the actual device behavior than the behavior documented in the datasheet. Creating an accurate mock is, sadly, often just not possible with the documentation available. And with the quality of mocks being bad, you end up testing the datasheets against the devices instead of testing the kernel code against the devices.

Serious question: how many CVEs are due to bugs in code that falls into this hard-to-test category?

That's exactly the mindset why baremetal code is so horribly buggy.

Creating a mock is still better than not creating a mock to test against. With the mock you can create simulations, unit tests, coverage tests, static tests, formal verification.

Without mock you can just test on actual HW a few selected test scenarios you think of.

Emulation: The lack of qemu support for arm chips is mindboggling. avr/atmel is quite good in contrast. renode is far better.

> Creating a mock is still better than not creating a mock to test against. With the mock you can create simulations, unit tests, coverage tests, static tests, formal verification.

You did not understand what i wrote. With hardware, its just damn difficult to get accurate mocks. If the mock is wrong, all tests relying on it are not meaningful.

If you ask me, QEMU doesn't even properly support emulating x86. Its just "good enough" to run common operating systems. You could try improving the support, maybe then you will realize how just damn difficult and undocumented this shit is, and you would understand why nobody wants to hear your crappy unit test evangelism.

Well, my company is happy to test baremetal fw on sw, as other better companies do also. Ditching tests is a lousy argument, only coming from people who have no idea how to test difficult targets.

Of course you have to patch qemu, gdb, renode and valgrind if something is missing. Eg the recent apple M1. But this is only needed for emulation, and we are talking simulation here.

> since it depends more on the actual device behavior than the behavior documented in the datasheet.

That doesn't sound true at all tbh. I would imagine a huge amount of code is just generalized data structures and algorithms. I doubt many CVEs (<1%) rely on an accurate model of the underlying hardware.

Spectre and meltdown can be seen as caused by the underlying hardware. But those are in that rare 1% bucket and I doubt a more accurate model used by unit testing would have stopped these anyway.

Unit testing coverage is a development practice, not really a defense.

Testing, coverage, static analysis, methodologies, etc... are a way to make prevent things from going wrong in the first place, defenses are here to protect you if things go wrong. We need both.

The article focuses only on the "defense" part, the development practices could make another article.

Note that testing and defenses can go together: test to make sure all paths are taken before shipping, and see if defenses trigger. Fuzzing in particular takes that approach.

Unit tests are not the only way, or the best way, to test software.

Sure? The note is that it is not a way here.

Because a kernel is not a user-land?

I agree with parent that unit testing privileged bare metal code is not always possible, at least not without spending 10x-100x dev time on writing tests.

Example: anything that directly accesses MMU or caches.

On the contrary, simulating MMU or caches are trivial. You can create a memory subsystem mock in a few hours.

IO, assembly and interrupts are much harder.

How would you create a mock object for something that modifies the memory layout of the unit test while it is running?

> How would you create a mock object for something that modifies the memory layout of the unit test while it is running?

There's two ways I can interpret "the unit test".

1) "How would you create a mock object for modifying the memory layout of the code being tested?" That's just a routine that shuffles around some bytes. Or marks parts of the address space to fault so it can simulate the contents.

2) "How would you create a mock object for modifying the memory layout of the code performing the test?" That wouldn't be a test. The code performing the test isn't inside its own simulation.

You create your mmap's on init.

The MMU is just a simple 3way array lookup, as a Unicode table.

Unicode?

Perhaps there was an /s missing in parent post? Implying that unit testing is often prioritized more by management, over checking security, configuration and tools like this one.

I can just imagine the technical and political difficulties in adding proper unit tests to the kernel now. The C model is "everything just imports and calls whatever it wants". It's great for speed, but for good unit testing you actually want some of that "AbstractFactoryFactory" stuff we make fun of in Java. For instance, take any module of the Linux kernel and try to see what it would take to test it without importing any other kernel code. Then, if you have a plan for that, now you need to stub out the rest of the kernel. Since every module imports dozens of files from other modules, and calls dozens of functions from said modules, it's not very pretty.

Reply on news.ycombinator.com