Live-migration support in FreeBSD and bhyve coming
gyptazy.com
gyptazy.com
Quite a bit was changed between "illumos bhyve" and FreeBSD upstream. We in illumos decided to take a slightly different approach in how we approached the machinery for saving/restoring the state of devices emulated in-kernel, which is a necessary part of doing such a live migration. FreeBSD was first to add that functionality, but we (I) felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware. It was at that point that illumos "soft-forked" the kernel VMM component so we could take a different approach. While the two kernel VMM components are very similar, and much of the bhyve userspace emulation (a counterpart to propolis) will run on either OS, there are some key differences around instruction emulation and those kernel-emulated devices which prevent them from being drop-in replacements for each other.
All that said, I'm happy to see FreeBSD making progress on this front. They've been a friendly peer in this space, and both projects benefit from efforts in this area.
> The "good" ones today are looking exclusively at external properties rather than internal ones.
To provide a little additional context here: This effectively was an external property of the system (just to the userspace VMM). One of the significant challenges for those implementing a feature live migration is how you'll deal with multiple version of the software (older and newer) interacting in the future. A primary use case for live migration is moving workloads around to permit software updates, making that multiple version issue even "worse".
Abstracting the implementation details of those in-kernel emulated devices was helpful in two ways. Firstly, it reduced (hopefully) the chances that a later change to the internal implementation would be visible in the exported/imported state. Secondly, that the act itself of doing that state abstraction required a much more intentional and thorough examination of the emulation logic itself, resulting in a cleaner and simpler artifact.
So while it may seem like it was driven by a desire to take pride in the craft (undeniably _a_ factor), the main motivation was avoiding a versioning mess in the future.
> Thank you for the comment.
My pleasure!
I love FreeBSD. But I reluctantly had to migrate one of my home servers to Ubuntu Server because I needed Docker support as well as ZFS. Ubuntu Server, compared to FreeBSD, is a complete clusterfuck of an OS. And I regret the effort I spent migrating.
I did, for a period, run some other distro with DKMS but that was a massive pain to keep up-to-date due to ZoL’s piss poor documentation and messy build tools.
This was quite a few years ago though. Long before even FreeBSD synced with ZoL for OpenZFS. So the problems I encountered have likely matured tenfold since.
But if I’m going to the hassle of installing a new OS then I might as well install FreeBSD again because I just prefer it to Linux (nothing against Linux in general, my laptops and PCs all run Arch and I’ve made a career from supporting a variety Linux platforms)
[1] https://github.com/freebsd/vm-bhyve
On my desktop I also use ZFS with kmod on cachy and updates have been seamless so far.
Wow it’s scary how quickly time goes by!
My current setup for Docker (Podman really in my case):
Proxmox -> Linux VMs -> Podman containers inside Linux VMs
My plan: FreeBSD's bhyve hypervisor -> Linux VMs -> Podman containers inside Linux VMs
So basically just ditching Proxmox for FreeBSD/bhyve. Then at long last I'll be, again, systemd-free.But maybe you’re right that it’s worth reconsidering now.
[0]: https://docs.freebsd.org/en/books/handbook/containers/
[1]: https://podman.io/docs/installation#installing-on-freebsd
So the project is alive and it seems more active than previously.
But I guess we are at the point security is important and hardware ( CPUs ) is relatively cheap?
Just general angst some people were expressing for publicly exposed resources. The common wisdom just a few years ago was that throwing an internet facing nginx or haproxy on an up-to-date LTS linux with some firewall debouncer is as rock solid as you could get. And instead, you should worry about security in your own upstream app, not Linux's TCP sockets or nginx's regex path matching.
It indeed is! Looking forward to the next release, which should happen "soon", as 6.4 is already 4 years old.
Linked from today's https://www.reddit.com/r/freebsd/comments/1xwianz/comment/19...
Unfortunately, it never made it into the code base, but they were the pioneers!
But ... OpenVZ in the past had 'live migration' - so it would be nice to implement that feature into FreeBSD Jails to have 'high availability'.
Also neither of us spotted that I wrote 2025 instead of 2005 but I can't edit anymore. Moral of the story, don't write comments at 7am.
FreeBSD had Jails containers since 1999.
Linux joined the party starting with on 2014.