Running FreeBSD on OS X using xhyve, a port of bhyve | Hacker News Reader

Running FreeBSD on OS X using xhyve, a port of bhyve

dan.langille.org

185 pointswhalesalad39 comments

I've been using this setup for years and glad to see some info here on HN about it. It's very fast but more importantly for development it really does a great job for porting to the real thing. I've never built anything in Xhyve that fails on a real install. Most of the differences I've seen are in vmnet versus like an e1000 device. I don't believe it does full duplex as per the xhyve docs (its a TODO under virtio_net) and on the MacOS side there is no Tap device like with Hyperkit. Despite that web pages can be served quite fast, a lot of ops are a stub to mmap on the host. NFS runs really well as a native format to share between guest and host. Heh, I've left it on for a couple of months accidentally and was still running like a top. My only real gripe is NTP drifts when the host goes to sleep and there is not an out-of-the-box solution for that like say VMWare. Someone has gone part of the way with getting e1000 for FreeBSD with Hyperkit it just needs to be looked into more, but this is a good explanation- and explains the performance differences: http://thebsdbox.co.uk/a-peak-inside-docker-for-mac-hyperkit...

I hope xhyve will become a first class supported feature that gets shipped with macOS. I see recent commits from Apple engineers https://github.com/machyve/xhyve

I've played with xhyve and dockers version of it -hyperkit-, too. Even wrote a little bash script to help with creation, start and edit of VM config files for it. If anyone's interested.

https://github.com/Ozymandias42/xhyve-manager

AFAIK it’s not a port of bhyve but a Port of the bhyve userland tools that tap into the MacOS hypervisor framework.

I wish there was a simpler approach to the hypervisor framework. I normally avoid using any third party software that wants to taint the kernel and third party hypervisors fall into that category. There is parallels desktop lite which can be installed from the App Store but it’s extremely limited (and paywalled) for something that I suspect it’s just a pretty interface to the hypervisor framework.

Correct, xhyve is built on the macos hypervisor framework.

I've had good experiences with it running linux guests, it took a bit of tweaking to get networking setup, but other than that it's been solid.

>running linux guests

Can you point me to the resources you used? Thanks.

the xhyve repo came with a small Linux setup

time (git clone github.com/mist64/xhyve.git; cd xhyve; make; ./xhyverun.sh) w/ "sudo halt" pasted into the resulting shell: 14.85s

It's been a while, but this is the startup script I'm using:

https://gist.github.com/wiredfool/0e1aa099a29f562cdc3ba30c36...

The hard drive images are literally a qemu image for linux kvm that's been qemu-img converted to a raw format, then the kernel and initrd were extracted, and stored in the mac file system. This image was originally on my linux box for a while, before I needed to go mobile on the laptop with it.

How is IO performance compared to Vbox? I've seen some old benchmarks indicating that xhyve IO is like 4x slower..

I'm seeing about 15% slower raw speed than the bare drive, dd if=/dev/zero of=tmp bs=1M count=512 is giving me between 68 and 250 mb/sec on a 2013 MBA with a nearly full disk, where in the terminal I'm getting between 78mb/s and 290mb/s.

ymmv, of course.

There was Veertu Desktop, which used Hypervisor.framework, but they have pivoted into CI for iOS. At some point it was open sourced, but it seems to have disappeared from their GitHub account. At least someone seems to have made a copy before:

https://github.com/tithomas1/vdhh

Thanks for the link, I was wondering what happened to veertu desktop. It was a neat project.

QEMU now has ‘hvf’ (hypervisor.framework) available in master as an accelerator, I think it was a summer of code project.

Wish I could use it, but the current implementation calls the xgetbv instruction which is not available on my Westemere EP processors.

All MacOS/QEMU needs is PCIE passthrough and you could have an amazing setup with thunderbolt accessories.

Was not aware of this! Thanks for sharing, this could be a great breakthrough for interfacing with hvf machines. xhyve is sorely missing an accessible management cli.

Mmmm eGPU + Passthrough would be sweet. Imagine gaming or using GPU heavy applications on a Windows VM with dedicated GPU and without the need for a separate set of monitor and input devices like Looking Glass does. https://github.com/gnif/LookingGlass

> QEMU now has ‘hvf’ (hypervisor.framework) available in master as an accelerator, I think it was a summer of code project.

It was not a Summer of Code project, there was (is) a company called Veertu that did the work who sold a virtualization product for running Linux and Windows on macOS on the App Store. Because it used Hypervisor.framework, it could be sandboxed and could be on the Mac App Store.

Google forked Veertu's GitHub repo, adding a few bells and whistles for use in their Android emulator.

Finally, the work was merged upstream to QEMU.

What's the overhead of running in xhyve? Is it like a Linux kernel container, or more like KVM, or what?

It's a virtual machine, not a container based system. So it's going to perform somewhat like newer versions of VirtualBox I'd imagine. I'm going to take this for a test spin and if I get time I'll post back my benchmarks here.

Isn't running any BSD on OS X redundant?

No more so that virtualizing one linux distro on another. Or running a Windows VM on Windows....

And let's be honest, while macOS has a lot of BSD in it, it isn't FreeBSD.

For example: Pre-APFS, imagine running any software with a custom data store using file system holes. Definitely something you'd rather do in a FreeBSD VM than on OS X directly.

Not at all. I used xhyve to run FreeBSD on my mac laptop, because I want to have a dev environment that matches production (except for performance wise). Mac OS's BSD userland is sort of similar to FreeBSD, but there's plenty of differences, and although it took a while to get xhyve working (and before it was available, I used virtualbox, which also took time to setup), the gains in being able to try something and know it will work on production, instead of trying something and having to try it again were worth it.

I've been moving off Ubuntu in production since it adopted systemd, and a local FreeBSD VM is essential for developing & testing the provisioning and deploy scripts.

nice nice

It's been macOS for the last couple of years.

Why run FreeBSD, a sibling of macOS? It's kinda incestual.

Thanks for sharing the scripts. The instruction is intuitive -- well organized and contains a very detailed description of the virtual disk part, very neat comments. As a long-time user of OS X and homebrew, I must say I become very interested in your solution -- running PostgreSQL natively on OS X, or macOS.

However, it makes me so confused that if it is your intention to run something heavy in a consumer-level Operating System. As far as I know, macOS has a newly-added but slowly-performed file system, named APFS. It might be good enough for single disk request but not good for the concurrent requests (like 4; ; ) -- I am not questioning/blaming about the old port design of either xhyve, OS X, or even hardware architecture SCSI, but that's where we can start now.

Anyway, there is a newly-adopted storage technology named NVME emulation which might help a lot in your case -- not only it handles with lots of disk requests at the same time (65535 queues and 65535 commands per queue), but also the architecture makes it possible to program the controller code in software, especially in user world. I don't know all details of NVMe but it works well in my experience -- both of consumer level and enterprise level -- it brings smooth disk latency in the virtual machine even under a very high bandwidth (50Gib transfer in nearly half a minute, can you believe that?). Do you know if it is possible to use NVMe emulation for faster disk access in xhyve and accelerate the PostgreSQL execution natively?

- SCSI: https://en.wikipedia.org/wiki/SCSI - NVME: https://en.wikipedia.org/wiki/NVM_Express

No idea about xhyve in particular, but bhyve got the NVMe support in FreeBSD 12.0.

>However, it makes me so confused that if it is your intention to run something heavy in a consumer-level Operating System

I assume this is to build a development environment. Especially if you're working on a laptop, on a client-server app that needs to talk to a Postgress (or other) database, running the database in a VM so it goes where you go is a pretty common setup.

It is true and that's probably the most unclear part of the original blog. I think the author has given some kind of advice -- use a vnet based on host-bridge -- not sure why he just commented out from the script. In my understanding, if I am working on a laptop, I can just set SNAT rules which allow traffic from a private network to go out to the internet. Given that the number of connection won't be large (no more than 400 connections, or 500, a usual number of open file for one process), it is acceptable that the new connections can wait until a period of time.

It would be great if it was possible to buy MacBooks with FreeBSD instead of OSX, which is lately bigger and slower compared to FreeBSD.l, especially due to the useless graphic effects and the bloatware installed (iTunes as example)

I don't think iTunes comes pre-installed these days. It gets updated so frequently they decided to just prompt the user to do an install on first boot. Certainly if you do a fresh install of Mojave over the wire, as I did, it's pretty clean. No iTunes, Photos, iMovie, Pages or any of their pro-sumer stuff by default.

Good ninja edit, was about to tell you that you're wrong! iTunes most certainly does come pre-installed on 10.12, can't say about 10.14

I realized I was being a bit cavalier and did a bit of checking. Been a while since I bought a fresh Mac, but last time I must have done a fresh install off the wire straight away and forgotten about it. My bad.

I'm pretty sure iTunes still comes preinstalled in Mojave. Not to mention Chess, DVD Player, Home, News, Stocks, Books, and a fair amount of other crap.

I have a post-install script I always run that does sudo rm -rf on all the apps I don't like. Has no ill effects in all the years I have been using it, and makes my system feel clean.

I have it, and didn't intentionally install it. Maybe got installed after logged in with my AppleId? I'm not sure how it got there, but it did.

Reply on news.ycombinator.com