Hacker News Reader

Comment by amluto | Hacker News Reader

It's never been clear to why very simple unikernels are better than running essentially the same code as a normal user program.

View on HN

All the replies so far miss amluto's point: A unikernel running on a hypervisor is a process, just with an awkward syscall interface and inflexible memory management. Especially when the hypervisor is KVM which runs VMs as regular Linux processes, this is very clear.

> A unikernel running on a hypervisor is a process, just with an awkward syscall interface and inflexible memory management.

...and with no way to run ancillary scripts and cronjobs e.g. log management, backups, security tools... Nor ssh into the VM to inspect it and run debuggers or monitoring and metric agents.

Essentially you are giving away all the practical advantages of modern OSes. There's a reason why the usual famous tech companies don't use unikernels, generally speaking.

Not the case, Google for example are looking at using the Linux Kernel Library: https://netdevconf.org/1.2/session.html?jerry-chu

VM interfaces are awkward due to emulating hardware so that they can run existing operating systems. A VT-backed unikernel interface does not need to inherit this awkwardness.

As for memory management, in the Solo5/ukvm case the amount of memory you grant to the guest (monitor process) is essentially the maximum you expect the guest to need, ever. So it's similar in terms of overhead to many usermode allocators which just mmap() a huge block of memory at start and then hand out to the application from that.

That's how a regular KVM works as well (mmap a physical memory block) and the mismatch comes from the fact that all the regular OS memory management mechanisms -- reclaim, swap, page cache, etc. -- are all broken by the abstraction provided.

FV guests can similarly have a balloon driver, that doesn't mean the interface isn't awkward.

A user process running on an OS running on a hypervisor has certain overhead, so you can either remove the hypervisor (native containers) or you can remove the OS (unikernels). Arguably the hypervisor has a smaller attack surface than an OS.

Also, there's a small set of programs that don't get along with the abstractions provided by OSes and run better on a virtual hardware abstraction instead.

You realize that the guest OS _does_ things right? It's not just overhead. You can't just remove the Linux network stack, you have to replace it with something. (And the Linux network stack is pretty damned good at what it does.)

The intrinsic overhead of the OS (i.e. costs of mode switching) is much smaller than you might think.

It's true that some programs don't really want an OS getting in their way, but a modern operating system provide decent facilities for getting _out_ of the way (e.g. O_DIRECT for block I/O, pinning tasks to cores) and that's usually what those things use.

This is backwards.

A userspace process on x86 has a certain amount of overhead, namely the overhead needed to switch between it and the host kernel. This takes the form of SYSCALL/SYSRET (mostly) for system calls and interrupts/IRET (mostly) for interrupt returns. SYSCALL and SYSRET are quite fast. interrupts and IRET are slow.

A VM guest needs to do a VM exit (which is quite a bit slower than even an interrupt, I believe) and then fix up a bunch of host state that doesn't get restored by the CPU. Then, to get back to guest mode, it needs to manually restore some guest state and do VMRESUME.

Regular user mode wins hands down.

To be fair, unikernels get to directly create page tables, which is potentially very useful. But, for the simpler applications, it's probably not.

(I more or less maintain the glue between x86 Linux user code and the kernel. I do, in fact, know exactly how this works.)

The number of calls probably wouldn't be the same, though. A process might make hundreds of system calls while a unikernel might make one hypercall to do the same work (because virtio has batching).

Anyway, I don't personally think reducing system call overhead is interesting but it seems to be one of the arguments unikernel people make.

I see no reason whatsoever that you can't do virtio without virtualization.

Unfortunately containers aren't considered secure enough for malicious users while VMs are.

Solaris/illumos Zones and FreeBSD Jails most definitely are considered secure for malicious users. And they've been secure for more than a few years.

Neither are most unikernels. Especially the weak points are file system and network handling. Generally the security of VMs is enforced by hardware and improved by limited attack surface - few drivers to audit, limited communication APIs. You can do the same with general purpose OS by cutting options.

I'm not talking about unikernels. I'm talking about the VMs that isolate them from each other. The VM isolation mechanism is considered secure enough to isolate malicious users, while the container isolation mechanism isn't (at least Linux-based containers).

There's a whole zoo of "little operating systems" out there that more or less have unikernel-like properties.

Some more closely meet the definition of a Unikernel, others are more like stripped down Linux, others are more exotic OS's that are nothing more than a network driver and disk driver in a virtual machine, such as Solo5.

I got Bootrino (see elsewhere) to boot all of these types of operating systems including embedded Linux (Yocto), Solo5, TinyCore, Alpine, and some other weird run-from-RAM OS configurations.

I got MirageOS running on all of the above, so unikernels can fit within other OSs or live on their own.

By their nature, unikernels are extremely well suited to running on top of a simple and well-defined interface. By extension, such interfaces are easy to audit and secure.

Why is this better than running as a normal user process? Look at the complexity inherent in securely sandboxing applications using existing interfaces -- the Linux system call interface amounts to hundreds of system calls, with different mutations for different architectures and many calls multiplexing even more operations (eg. ioctl()) the numbers grow even higher.

Unikernels give us a shot at redefining the process interface, while still being able to run on existing infrastructure (host OS or hypervisor). This means we can work toward a system where the application (unikernel) has least privilege by default.

With Solo5 [1] we're working towards defining such an interface. We have a way to go yet [2], and are still very much in the "early adopter who knows what a unikernel is" stage, but the possibilities are exciting, not only from the application security point of view.

[1] https://github.com/solo5/solo5 [2] I'd be the first to say that the current Solo5 interfaces are suboptimal, but we're working on it: https://github.com/Solo5/solo5/pull/201

How is this any different from running a process with a very short whitelist of system calls? Compare to the original linux seccomp, which only allowed read, write, exit and sigreturn.

Yesterday there was a post about a coming linux kernel feature, kernel TLS sockets. (https://news.ycombinator.com/item?id=15164568)

One of the justifications: it removes the need to copy data between kernel/user-space.

KTLS strikes me as inelegant. Socket programming requires you to reason about your frames on the wire. Proto-socket features complicate this.

A unikernel approach would allow us to eliminate unnecessary copies, and could be done with more elegance than KTLS.

The Linux kernel is a dumping-ground, kitchen-drain of all features (20 megaLoC with no end in sight) because of their inability to isolate concerns and provide fast IPC.

And yet able to capture the server and phone market so completely. Best isn't always best.

Shared memory and busy spinning work normally when you need that latency (high freq. trading for instance)

You also forgot about security in this IPC which is very much not a priority for Linus. With good reasons - security community has no consensus there.

Is the obsession with avoiding memory copying really justified? I'd like to see some data. Is this really a bottleneck in modern systems? Is TLS a bottleneck at all for most work loads?

IIRC Netflix found TLS to be a bottleneck when serving 100 Gbps of static content over HTTPS, but that's an unusual case.

The kTLS paper showed pretty small speedups, so maybe the real motivation is TLS hardware offload.

Memory bandwidth still matters, yes. Cache sizes are still not very big, and a copy uses more cache. Obviously you may have other bottlenecks first, but it matters once you are using 10Gb ethernet or faster, for example.

That's a good point. It's probably more a cache thing than a CPU cycles thing and that might not show up on a straightforward CPU profile.

What would happen if you used special instructions to try to implement a non-cache-polluting memcpy? Or is that even possible? It's been a while but I seem to think that modern CPUs have a variety of cache control instructions. Zero copy is great but it can be a PITA.

You have to use at least one cache line for memcpy. But you need more for pipelining the accesses probably to get good speed. Benchmarking memcpy though is complicated.

The smaller attack surface and the possibility to implement optimizations across boundaries (userspace vs kernelspace, for example) are the selling points.

I'm not buying into this myself as of yet but at least in theory this could lead to.an "application provably correct all the way down the stack".

What smaller attack surface?

If you compare the attack surface of a Linux user process that is very tightly seccomped to the attack surface of a whole guest VM (with all the awkward emulation needed), it's not at all clear to me that guests are better.

Better isolation and a cleaner interface. There's a huge hodgepodge of different ways user programs can communicate and interfere with each other, many of which are rarely used and don't make sense for most programs but have to be retained lest removing them break something.

In which case something like seccomp is a better vessel for these programs? VM interfaces have lots of surface area.

Quite the opposite. The Unix userland <-> kernel and IPC interfaces are enormous: hundreds of system calls, often overlapping, a dozen different socket types, multiple different kinds of magic filesystem interfaces (some of which take complex datastructures in arbitrary unstandardized text formats)... Whereas the interface between a VM and its host is pretty small.

seccomp is a Linux system for restricting a process to a small # of syscalls.

Sure, but the set you'd restrict to would look rather like the VM interfaces, only more awkward because it's had to be retrofitted on to what already exists, and often more permissive than it should be because a library assumes all syscalls are available (why would it not?)

The virtualized hardware interfaces also enormous. And more cumbersome to restrict than seccomp, and more cumbersome to use than syscalls.

I'm not sure I follow the logic in your harder-to-retrofit opinion. In unikernels the central idea is retrofitting the unikernel around the host interface and the application. It seems obvious to me that this would be more easier to develop and debug inside a standard process, than an emulation of bare iron, even if you would have to call it a different platform due to a radically smaller syscall interface.

> The virtualized hardware interfaces also enormous.

I don't see that they have to be? There's no reason for the host to offer e.g. a virtualized graphics card except to unikernels that specifically want to output graphics.

> In unikernels the central idea is retrofitting the unikernel around the host interface and the application.

I don't think I follow? The only part that's retrofitted is using an interface that was designed to host OSes to instead host an application, but it's a good fit because virtualized guests haven't been treated as trusted (particularly not in contexts like AWS), whereas for decades of unix history any local process was treated as more-or-less trusted.

> It seems obvious to me that this would be more easier to develop and debug inside a standard process, than an emulation of bare iron, even if you would have to call it a different platform due to a radically smaller syscall interface.

If we're talking about creating a new platform with a restrictive, strictly secured syscall interface, whether we talk about a hypervisor running unikernels or a kernel running processes is pretty much just terminology. I would think the actual interface specification would end up very similar to the current hypervisor/unikernel one.

The big advantage of keeping compatible with existing VMs is to provide a way to run legacy processes without compromising the security of new-style ones (and while making sure the incentives in terms of e.g. performance run in the right direction). If one component of your system needs to be run as a traditional unix process, that's fine - it gets its own VM in which it's running a traditional kernel and the actual process - but it can't interfere directly with your other components, and it pays for its own overheads.

I agree that VM interfaces don't have to be big, but they are by default just like the syscall one is. This argument was just to set the two at a similar starting point.

Historically Unix processes and VM guests have both been treated as untrusted on paper, but with not much scrutiny. There are frequent VM breakout vulnerabilities announced and there's wide agreement that this is a porous security boundary. The numbers are probably better for the VM interface, but it's in the same category.

Apparently we agree that in terms of exposed functionality, the end result is not going to be very different, if we restrict the VM interface vs use seccomp or similar.

A difference here is that there is already an easy and widely supported and understood system for this restriction for the process case. And processes are easier to debug, instrument, monitor and manage. And they don't require platform support for virtualization, which can be useful on eg embedded platforms, phone apps etc.

For the last point about running legacy processes, maybe we have a different understanding of what Unikernels are. If you have an unikernel application that uses a Rump kernel or MirageOS, you are not going to be able to run legacy processes in the Unikernel container in any case, there is not even going to be a kernel/userspace boundary. There is just one code image with everyting linked into it. So the traditional VM interface is not going to help with the legacy processes.

If you were talking about running regular VMs for legacy apps alongside unikernel images, then that's of course valid, but I don't agree that this is an argument for using the VM interface for unikernel images.

> A difference here is that there is already an easy and widely supported and understood system for this restriction for the process case.

There is? I see a bunch of ad-hoc, retrofitted efforts, which seem to be constantly playing whack-a-mole in terms of what they forgot to restrict.

> And processes are easier to debug, instrument, monitor and manage.

Yes and no. The kernel/process boundary can make debugging and profiling much harder when you need to do so across it. With a lot of unikernels you can run the whole thing as a userspace process for development purposes and debug/profile without worrying about that. And these days we have a lot of good tooling for monitoring and managing VMs.

> If you were talking about running regular VMs for legacy apps alongside unikernel images, then that's of course valid, but I don't agree that this is an argument for using the VM interface for unikernel images.

That's what I meant. If you have a mixed deployment of legacy components and new-style components, that approach lets you realise the benefits quicker for the new-style pieces and ensure there's an incentive to move components into the new style. Whereas if you just run a bunch of processes, then no single component can benefit from isolation until you've ported everything.

seccomp does not imply a whack-a-mole type of effort: you can have a very small whitelist of syscalls, and further restrict their parameters.

The debugging would be similar with a unikernel run as a seccomp process: you could debug it as a single process. I believe the BSD rump kernel is already used like this for debugging purpouses.

If you want to mix legacy processes and unikernels in parallel, you would still run the legacy processes on a different VM in the seccomp case.

Reply on news.ycombinator.com