If you're interested in "other things" that can be done with KVM, take a look at the Solo5 "hvt" tender: https://github.com/Solo5/solo5/tree/master/tenders/hvt.
297 karma · joined January 28, 2010
Speaking as a long-time user, over the last 10(?) years I've only ever needed to reach out to support@ twice or so, both times with fairly obscure issues that were promptly dealt with -- thank you.
It'd be a shame if the implied change to "community support only" for free accounts means that free users no longer have any direct way to contact support.
> - 65 were not guest exploitable
> [...]
Which leaves about 30 that presumably were guest exploitable.
Don't get me wrong -- QEMU is useful. As a "kitchen sink" solution that runs anything, anywhere, with any useful combination of emulated {devices,processors,systems}.
However, this is also its biggest weakness. Which is why Google and Amazon all run their own custom VMMs for their IaaS services.
The microvm machine type as described here is a great step to improve this situation. The next step in my book would be to reconfigure QEMU's build system to allow building a binary that only supports the devices provided by microvm, and nothing else.
Indeed. Nice to see that the cross-pollination is happening.
For folks interested in what can be accomplished with userspace VMMs, a very minimalist example is the Solo5 project (https://github.com/Solo5/solo5), specifically the 'hvt' tender.
The notable thing missing at the moment is ASLR and a more robust guest memory layout in general. We have an issue with a plan for that but it's a matter of balancing a finite amount of time/manpower and priorities.
The largest part of a monolithic kernel today is devices drivers, by far.
> Networking isn't trivial and I don't think anyone will be reimplementing the TCP/IP, UDP/IP, ARP, DHCP and more stacks without breaking in a few bugs themselves.
Sure. And then more people will use those stacks, and they will get better. The more the merrier, we have too much of a software monoculture anyway.
> The only real advantage I see for unikernels is that because all the hard work is done by the hypervisor they don't have to bother implementing device drivers
This. People continually underestimate the amount of work required to support the hardware ecosystem. This is also why rump kernels (note, not the same term as the unikernel known as Rumprun) are such an achievement, also very much underappreciated.
For the 80% case, on x86_64, I consider them more or less equivalent. KVM is used daily in anger to provide isolation (e.g. GCE, and now ChromeOS) and has been around much longer but you need to trust hardware virtualization which is a large attack surface on the CPU itself. Given what we've learned about CPU vulnerabilities over the last year, I wouldn't be surprised to find some lurking in the VT-x/SVM implementations.
Seccomp OTOH is difficult to use correctly for arbitrary/existing applications but exposes less of the kernel (depending on your metric, see our paper) and does not need hardware virtualization.
For the 20% case, where the stakes are higher (e.g. High Assurance), I would use something like Muen or SeL4 and run a disaggregated system on top of that.
Correct. Conceptually the same applies to all Type 2 hypervisors. Type 1 less so, but you could still potentially exploit Xen and you have all of the dom0 to play around in.
> The paper suggests using system-call filtering via seccomp or similar to block off this attack surface, but if you trust seccomp to have zero bugs, you might as well do traditional process sandboxing a la Google Chrome.
If you do that, you will have to do a line-by-line analysis of the code you want to sandbox in order to determine exactly which syscalls it's using. With the approach presented in our paper the developer does not need to care about this. For example, she can develop her MirageOS unikernel as a normal UNIX process and switch to a guaranteed-to-work-minimal-seccomp sandbox with a simple change of target (build-time configuration option).
But yes, you are now trusting seccomp instead of KVM. I believe in giving people the ability to easily make that choice.
> The problem unikernels solve is that traditionally [...] can't rely on eg. a shell or a debugger being available to arbitrarily move bytes around.
That part does not change with the sandboxing mechanism changing. The stuff that's inside is still only your unikernel.
To add to this, with the approach we're taking with Solo5 [1], hardware virtualization is only used as one possible sandbox/isolation mechanism. If you look into the code, you'll find that the abstractions presented to the unikernel by the hvt tender are much thinner than those presented to a guest OS by a traditional VMM (e.g. KVM/QEMU).
1. Regarding bugs: There is a lot of code in your monolithic OS kernel. There have been, and will be, a lot of vulnerabilities in that code. Various sandboxing mechanisms notwithstanding, the vast majority of processes running on your system can potentially use all of that code as an attack vector. Unikernels let you switch off access to most of that "host" code, and, through the use of a library OS, contain only the minimal set of libraries needed to run your application in the "guest". Regarding updates: Valid point, but no different from any modern application of substantial complexity, except that now you also have to update (e.g.) the library providing its TCP stack.
2. Unikernels don't make that problem any worse. In fact, if you run on something like Muen (https://muen.sk/), it'll mitigate a bunch of these attacks by giving you a less precise RDTSC at the subject ("VM") level.
3. I don't follow. Implement what?
So, I'm curious, am I just extremely lucky to have competent sysadmins as neighbours on the colo subnet, or is my IP address' reputation so good that it never gets affected by collateral damage?
Aside: I really like what the ChromeOS team has done over the years to advance the state of OS security for consumers, keep up the good work!
* For example, DoS-safe
As for memory management, in the Solo5/ukvm case the amount of memory you grant to the guest (monitor process) is essentially the maximum you expect the guest to need, ever. So it's similar in terms of overhead to many usermode allocators which just mmap() a huge block of memory at start and then hand out to the application from that.
Why is this better than running as a normal user process? Look at the complexity inherent in securely sandboxing applications using existing interfaces -- the Linux system call interface amounts to hundreds of system calls, with different mutations for different architectures and many calls multiplexing even more operations (eg. ioctl()) the numbers grow even higher.
Unikernels give us a shot at redefining the process interface, while still being able to run on existing infrastructure (host OS or hypervisor). This means we can work toward a system where the application (unikernel) has least privilege by default.
With Solo5 [1] we're working towards defining such an interface. We have a way to go yet [2], and are still very much in the "early adopter who knows what a unikernel is" stage, but the possibilities are exciting, not only from the application security point of view.
[1] https://github.com/solo5/solo5 [2] I'd be the first to say that the current Solo5 interfaces are suboptimal, but we're working on it: https://github.com/Solo5/solo5/pull/201
If the hypervisor sets the relevant VMCS control fields for the unikernel to cause a VMEXIT on (for example) a load of CR3 and any EPT violations and sets up the guest with a static set of read-only and executable (or in fact, with EPT you could even do execute-only) pages then there is no way the unikernel can disable these protections.
Having said that, I'm not arguing that running unikernels in Ring 0 is the best approach for security, just that it's not impossibly insecure.
With ukvm we're also looking into running all of the unikernel in (virtualized) Ring 3 with nothing running in Ring 0. However, this needs more experimentation since with x86 you can't (for example) easily handle legitimate exceptions (e.g. divide by zero) from Ring 3.
Minor point, but this seems to be a bit lost in the discussion: Generally 1 unikernel == 1 VM (or, virtualization-backed sandbox, the use of "virtual machine" brings too much baggage with it) == 1 application.
So, the attack scope for the class of attacks we're debating is equally limited to a single application, just like on a normal operating system.
And mmap(), mprotect(), ptrace(), ..., $syscall_du_jour(). It's not about fork/exec, it's about limiting the APIs available to the unikernel by default and by design.
Yes, you can do this for conventional Linux applications. But I wouldn't call it super easy, at least not for the vast majority of developers out there.
2) The same applies to any application, not just unikernels.
3) I completely agree.
I disagree. There's no reason such mitigations (not sure what exactly you're referring to) can't be implemented by the monitor process (ukvm in the Solo5/ukvm model).
I'd also argue that a normal kernel does not do any integrity checks on the code running in a user process, so the model is exactly the same.
> Even worse, the attacker could use it as leverage to infect other unikernel based instances of the app to gain some permanence against restarts by simply reinfecting when an instance goes down.
For that they'd need to break out of the virtual machine and into the hypervisor / monitor. Which is by no means impossible, but with careful design of unikernel-specific monitors can be much reduced. Of course, I'm by no means suggesting you should back your unikernels with a monitor along the lines of QEMU :-)
I'm not familiar with these. Googling "MOSDEF attack" suggests Massive Attack songs and "CORE IMPACT" various products by Core Security, I presume you're referring to neither? :-)
> Only executing native code is no defense at all. You're not even stopping attackers from the '90s like that.
> ROP is only a hardship until exploit code can find a way to allocate executable memory or hook the program's own dynamic execution (...)
Only executing native code loaded at deployment time, i.e. presumably from a trusted source. What I meant by "immutable page tables" is that after loading the unikernel image the set of executable pages is fixed, and all of those are read only. This is not something current unikernels do, but that's more due to lack of development time rather than any inherent implementation complexity.
If there is no way for the exploit code to ask the hypervisor for new executable pages or overwrite existing ones then the amount of damage it can do and its ability to persist in the running system is greatly reduced.
> Really, what you're describing here is the browser security model. When staffed by the largest, best security teams in the world, the browser security model almost works.
It's similar but not the same. The browser security model breaks down in a large part due to exposing way too many APIs, most of which are too complex to sensibly audit. Linux system calls have a similar problem, which is why you have to hand-craft things like seccomp profiles to match your particular application.
Unikernels (as used to implement network services) lend themselves extremely well to running on top of a well defined API / sandbox boundary, designed with security in mind. Again, we're not there yet, but the steps we need to explore in this direction are fairly clear.
In the case of a unikernel deployed on a hypervisor this is not the case, since there is not much else in ring 0 that you wouldn't already have access to from ring 3. Conceptually you can think of the hypervisor as "kernel space" and anything inside the unikernel as "userspace".
There are advantages to running the unikernel solely in ring 3 (eg. immutable page tables) however this is not a requirement for security.
A unikernel application which only executes native code[2], combined with a hypervisor / hardware that enforces immutable page tables and NX, AFAICT leaves ROP/JOP as the sole remaining attack vector for RCE. Further, each deployment of the application should be done via a new build (or at least re-link), which can employ the build-time ASLR Per mentions to mitigate ROP/JOP attacks.
I'd be interested in your thoughts on what other relevant attack vectors I've not thought of that would allow for a persistent compromise of a unikernel. Feel free to discuss here or on the devel.unikernel.org forum thread[3].
[1] Page 4 in this whitepaper from 2015: https://wiki.xenproject.org/images/3/34/XenProject_Unikernel...
[2] Does not contain a JIT or interpreter able to "eval()" new code.
[3] https://devel.unikernel.org/t/unikernels-are-secure-here-is-...