Run Bhyve in FreeBSD jails – The view from inside the forest (2020) | Hacker News Reader

Run Bhyve in FreeBSD jails – The view from inside the forest (2020)

neirac.byteswizards.com

98 pointsrodrigo97514 comments

I wonder how many nested VM inside jail inside a VM inside a jail... are possible from a resource perspective, so that I can still execute a shell

For a second you lit up my hope that there would be nested bhyve virtualization, but no. This is a different layer of virtualization/containerization. In order to have nested vm the correct cpu flags (vt-x) have to be exposed in the guest and I believe that such feature is unfortunately not implemented yet.

you can nest jails quite a lot. Although i have not tried it (and cannot find proper sources on the max limit of the amount of jails you can inside a freebsd system).

might be fun to give it a try though.

If FreeBSD has a maximum limit on PIDs that includes those running in jails, you'd probably run into that well before you run out of memory (especially if the goal is to run "as many jails as possible", you can make a whole bunch that effectively do nothing at all).

I am a fan of FreeBSD, and use it for a few tasks.. But I'm not at all clear why someone would want to do this? Does anyone have any idea what this gives you over just running a Bhyve VM outside of a jail?

Is it more secure somehow?

Before FreeBSD 12, you couldn't expose the bhyve components in a jail so you were forced to use the host as the bHyve hypervisor and/or jails. The new feature of the latest FreeBSD was a network stack within an jail.

With this stack, you can now assign an IP addresses to a jail. No longer requiring you to maintain a rule-set via the firewall while enabling you to operate as if the jail was the bare-metal FreeBSD host.

Knowing that the bHyve VMs are isolated within their own jails you can then hand those off to client as their own hypervisor space. With that they can run their own network, appliances, virtual machines while being isolated from anyone else with confined limits without knowledge of the host they run on.

This enhances public security as at any time I can axe the jail and everything is disabled; nor do I have to worry that the rule-set created will brick anything else. The client also has shell access in case of. And if a hacker breaks in to the bHyve VM, exploits out, the hacker are isolated to jail and not the hypervisor host.

Plus with the added bonus of ZFS, I can make snapshots or backups of clients jails with ease.

It can be MUCH more secure to run it in a jail; but IMHO that's not the point, but it could be.

There's also multiple types of security.

Running it in a jail allows you to do cool things, especially when coupled with ZFS. This is about running & managing VM's.

Imagine you wanted to 'restore from day X', or restart each day with the same config (i.e. testing Windows boxes for exploits), or you wanted solid A/B tests and MS keeps installing updates and moving your goalposts, or some funky abandoned software only works if you have the calendar set to 1993, etc....)

You can do all of these things easily with JAILS + ZFS.

But IMHO the best reason to run something in a jail is to isolate services.

On my file-server right now I have the following running in different jails:

Plex

Windows10

WindowsXP

my Git server

my SAMBA server

DHCP

DNS/PiHole/ad-blocking

IRC server

Mumble server

4 different FAMP servers for friends & family

I like to isolate each jail into the task it'll be handling. That how it stays organized in my mind.

ZFS send & receive + jails makes backups and restoring painless.

Interested. Now using LXD for the same setup more or less, but that needs the help of the host firewall with NAT forwards. Can you share or point to tutorials on having full network stacks in each jailed VM, that would be great.

I recommend using iocage for handling jails.

Here's a start for VNET/VIMAGE and jails:

https://iocage.readthedocs.io/en/latest/networking.html

BTW: VNET worked pretty stable but is "officially" declared stable for FB13.

You have to be brave to run bhyve. An update to my FreeBSD system caused a regression which made my VMs deadlock on boot.

https://jira.ixsystems.com/plugins/servlet/mobile#issue/NAS-...

I've been running bhyve on Dell for 10 years. You got hit by a problem, but any problem can crop up in any system any time.

You notice the problem was fixed, right?

It was one of several problems I have had with bhyve. When I tried, bhyve would fail to run about half of the linux distributions I thew at it.

Also, the more worry thing is that the project don’t appear to have any QA testing for linux guests. How is it that a bug which caused ubuntu and redhat guests to deadlock on boot got through the release cycle?

If there was any sort of QA testing for linux guests the bug would have been caught.

So my problem wasn’t necessarily the bug, but the apparent lack of QA processes.

I'm running debian and ubuntu, freebsd 11, 12, 13. I've run three variants of debian since sid. You pointed to ix systems, which are IIRC frozen on 11, when Freebsd is in release cycle for 13. Who should be doing the QA here? Ix or FreeBSD?

My TrueNAS Core install is saying "FreeBSD 12.2-RELEASE-p3".

My main issue with Bhyve is the lackluster networking performance, getting only ~1.5Gbps from VM guest to VM host (tested with iperf3) on fairly beefy machine. This really limits the usefulness of NFS shares.

Reply on news.ycombinator.com