Linux Hypervisor Setup (Libvirt/QEMU/KVM) | Hacker News Reader

KVM)

octetz.com

185 pointswilldaly62 comments

Probably my favorite feature of libvirt is the security and isolation features provided by sVirt. It applies a security policy via SELinux (or AppArmor) that ensures that in the event of a VM breakout exploit, the attacker can only access resources allocated to that VM. So it isolates VMs from the host and from each other. Really cool!

I’ve been using Proxmox community edition for my home datacenter.

It’s good to see what this article details into it and how some can pull all this together … for free (or for a small sum).

URL: https://proxmox.com/en/proxmox-ve/features

I was recently going to set up Proxmox as well, since I had used it ~7 years ago and it was working well. But I ran into an expectation that I have a drive dedicated to putting ISOs on, in addition to the OS install and the virtual machine storage. I was trying to set it up on a SFF machine and was limited in what I could put into it and I just kind of noped out of it. I couldn't install any VMs until I did this.

I don't recall this being an issue before.

I ended up re-installing TrueNAS Scale beta, which I had installed and tried, but ran into a problem. Ended up finding the solution on the forums, something related to bridged or VLAN interfaces not coming up.

In the end I probably will just install Ubuntu and put Ganeti on these boxes, since that's been really reliable for me at work. I was actually hoping to try out Proxmox to possibly move to at work.

What made you think that you need a separate disk just for ISOs? They can be uploaded to and stored on the default directory storage.

I snapped up an old Dell T710 2-socket, 12 CPU, 24-thread Xeon rack, PERc 6 RAID for $100, populated with cheap Seagate 2TB IronWolf HDs, leftover 1033Mz DDR3 to 96GB for 3 RAID HD groups (RAID 1 for OS, RAID5 for data, and JBOD for VM images.)

Has a pfSense/FreeBSD VM for a gateway and a bunch of VMs on a detached bridge. Also have several more bridges for test VMs for real-world airgapped network setup.

8 different OS distros. Also am tweaking a MIPS and a M1 QEMUs outside of Proxmox .

runs very quiet.

so now it’s my home datacenter.

I have something very similar, but an Optiplex.

One thing I had to do recently though is swap pfsense for Untangle because of drastically better FQ-CoDel implementation in Untangle.

I went over to Untangle website for any data sheet but was unable to get one immediately (have to fork over everything but your youngest child, so to speak in order to get it.

Not sure why would one even do that.

:-(

I don’t want to sidetrack this in to an Untangle conversation, but yes: it’s fundamentally not free as in speech while also being supported as though it’s just a thin shell over Debian.

However, the YouTube breakdown by Battle(non)sense was conclusive (and is confirmed in my case in undeniable ways) so I switched.

(I think) I ran into the same or similar problem you did - there's a partition for the debian OS + iso images of a fixed size (df shows it as /dev/mapper/pce-root -> / ) and all the reallocatable storage used by the VMs comes out of the rest.

Thing is it's hard to re-allocate this partition, and that's where all the iso images, and the backups go.

I tried to migrate a vm and boom - it filled up.

I tried to figure out how to reallocate it, but in the end, I found this (sort of hacky) solution documented:

https://pve.proxmox.com/wiki/Logical_Volume_Manager_(LVM)

basically:

  # lvcreate -n vz -V 200G pve/data
  # mkfs.ext4 /dev/pve/vz
  # echo '/dev/pve/vz /var/lib/vz ext4 defaults 0 2' >> /etc/fstab

you migrate the stuff from /var/lib/vz to the new location and you have 200G for iso images/etc

Why this has to fight for storage from the root system, or why it's not easily expandable is a head-scratcher. I think it might be better to have a dedicated boot debian partition, a resizable/easily growable pve partition, then storage that can be allocated to vms/containers.

but yeah - that stopped me in my tracks, and when I'm doing it on a home machine you just don't have the time to chase this stuff down.

Yep. so glad that i experimented (over a week) with Proxmox install over an existing Denuvan (Debian SysV init) install.

Made more sense to have a second drive or second but largest LVM to play with and store all things VM.

So confident that I disabled the Proxmox 'local' drive.

If you install Proxmox VE the next time and are not happy with the default layout, check out the parameters for LVM [0]. I personally prefer to install on ZFS as all ZFS datasets share the same free space unlike with LVM.

If you want full control of the disk layout and maybe want to do some more custom stuff, consider installing a base Debian and then install Proxmox VE on top of it [1].

The PVE installer makes it easy to get a system up and running quickly but is opinionated what the disk layout should be. That of course might not be working for everyone.

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#advanc... [1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_insta...

What do you all use for quick one-command (non gui) new vm bringup?

I haven't had much luck with terraform with libvirt, and I miss Vagrant but don't want to lock myself into its syntax. Maybe I should just start using it again but I feel like there has got to be a good libvirt-based way.

Provisioning new test vms with userdata embedded into an kickstart-or-whatever iso that is provided to libvirt also seems clunky. docker-machine is no longer maintained so I can't use it as a vagrant replacement to just bring up sshable hosts. Multipass?

Curious as to how others have solved this.

virt-install or virt-builder.

I have some extremely clunky 50 line shell scripts around virt-install presently but was hoping to find something a little more streamlined (like the usual Vagrant or docker-machine workflow).

Do you just manually run a huge virt-install command each time?

I usually use virt-builder actually. But it's like 3-4 command line options for virt-install, not 50.

Have you looked into Weaveworks Ignite?

https://github.com/weaveworks/ignite

I was playing around with this a while ago, using cloud-init and virt-install. But I found it to be quite cumbersome if you don't have some kind of metadata service that can provide cloud-init with the user-data, ie. not using the nocloud provider.

I started looking at this mock EC2 metadata service, but never did anything with it: https://github.com/sjjf/md_server

Ninja-edit: It seems like virt-install rencently abstracted the whole NoCloud-provider, this changes a lot of things: https://blog.wikichoon.com/2020/09/virt-install-cloud-init.h...

We implemented our own mock ec2 metadata service for KVM cloud-init, probably the most fun project I've worked on.

The tricky part is intercepting the requests and redirecting them to your service

You mention Terraform, I'm using https://github.com/dmacvicar/terraform-provider-libvirt quite heavily and it works a treat.

That's the one I've had suboptimal experience with. Maybe I need to learn it better. What kind of storage are you using it with? I ran into issues using file-backed trying to keep everything in tf.

I’m copying qcow2 volumes with it to file based storage and some Ceph RBD as well, I mean AFAIK it just talks to the standard libvirt volumes interface so anything you can do with that you can do with the Terraform provider.

I've had luck with Proxmox and this Terraform provider: https://registry.terraform.io/providers/Telmate/proxmox/late...

Other than a working Proxmox install, you'll need to create a cloud-init template, which is documented here: https://pve.proxmox.com/wiki/Cloud-Init_Support

Just wanted to chime in and say that, for my purposes, Multipass has been perfect for work and personal use.

https://multipass.run/

(For others, it's a simple-to-use CLI that provisions Ubuntu instances.)

virt-install with kickstart or preseed files and VMs attached to the serial console, like

virt-install -n centos -r 1024 --cpu=host -l http://mirror.cogentco.com/pub/linux/centos/6/os/x86_64/ --disk path=/var/lib/libvirt/images/centos.img,size=10,bus=virtio --boot kernel_args="console=/dev/ttyS0" -w network=internal,model=virtio --initrd-inject=centos.ks -x "ks=file:/centos.ks console=ttyS0" --graphics none

Setting up the serial console requires some configuration on the host part, too, unfortunately, depending on the OS.

vagrant with the vagrant-libvirt plugin. The roboxes project provides pre-installed and configured vagrant boxes for allmost all distributions. Its really just a matter of seconds to bring up whatever distribution box you like. Even pre-configured and sysprepped windows images exist.

I use qemu and never really understood the need for any further abstraction such as libvirt over it's cli usage. Can anyone tell me what there is to be gained other than needing further setup, packages and systemd services?

libvirt is mainly a consistency wrapper. So you can use kvm or lxc.

As for what it “gives”, I suppose I’ve never migrated a VM outside of libvirt, though I’m not 100% sure if that’s not possible with plain old qemu+kvm

I am not an expert but libvirt is not only for qemu but it supports various virtualization technologies such as Xen, LXC, VMware. So if you decide to stick with QEMU then I guess you don't need to use libvirt.

This is a blog post I wrote about the KVM userspace architecture, it goes into why Red Hat uses Libvirt in its virtualization products.

https://www.redhat.com/en/blog/all-you-need-know-about-kvm-u... (fixed now)

Getting a 404

https://www.redhat.com/en/blog/all-you-need-know-about-kvm-u...

It's not just about abstraction. VirtIO is significantly more efficient than SATA emulation, and as it is built into the Linux kernel it just works. There is also a Windows driver package that adds VirtIO support, but it's a bit tricky to get it to work when porting in an existing Windows VM. VirtIO also makes it possible to do USB relay.

VirtIO is very usable via QEMU, without libvirt (naturally, because in the configuration described in the article, libvirt is just calling QEMU). It is usually as simple as `qemu-system-x86_64 -drive file=/path/to/my/disk,if=virtio`.

There's more cool stuff coming in this area too. For a long time there's been the virtio family of protocols for shuttling IO to something outside QEMU to handle. Originally that was always KVM and the implementation is called vhost. Then later it became clear that these same messages could be sent to another user space process to handle instead (called vhost-user). These work great for creating virtio devices in the guest. But operating systems like Windows don't have virtio device drivers in-box, so it's a little annoying.

Recently, a new protocol to replace virtio has been defined. It is modeled on vfio ioctls and currently only can forward to another user space process, so we're calling it vfio-user. With this protocol, it's possible to emulate any PCI device rather than only virtio devices. Projects like SPDK (what I work on) can now use this to present fully emulated NVMe devices into guests and back them with whatever actual storage is available (a file, something over the network, a real NVMe SSD, etc). This allows an OS, including Windows, to boot from the virtual disk using it's in-box NVMe driver. This hasn't quite made it into a QEMU release yet, but it's close!

I'm like you and use the qemu command line. But libvirt does make it easier to define cpu groups, thread affinities and the like and without needing to be root to do so. Something like this. There are other ways to accomplish this I think without libvirt, but it gets a little hairy.

https://www.reddit.com/r/VFIO/comments/9iaj7x/smt_not_suppor...

When I started using virt-manager I started there because I was specifically looking to replace Virtual box due to having kernel modules for VB sometimes block kernel upgrades. So I guess I was looking for a GUI since I was on a desktop anyways. (Just have some dev VMs so I don't have to clutter my desktop with running MySQL etc for some projects and so I can match the distro of the production server)

So I guess, it worked, didn't look into raw qemu. I also use it infrequently enough that I would have to relearn the CLI all the time.

It's been probably a decade since I've used libvirt, but I can say why I like using ganeti:

- I can get a list of the host machines in my cluster and how much memory and storage they have available.

- I can easily move VMs between hosts if I want to evacuate a host for hardware/software/firmware maintenance.

- It has the ability to set up DRBD backed VMs and live migrate between the host nodes.

- List what machines are running and on what hosts.

- Start and stop commmands don't require me to remember the settings on individual VMs, the qemu commands that get run are something like 700 characters long.

> - Start and stop commmands don't require me to remember the settings on individual VMs, the qemu commands that get run are something like 700 characters long.

Shouldn't be the case. I put my VM args into a script, but it's only like 20 lines?

Actually, looks like I under-estimated by half. One of the random VMs I run has a command-line 1361 bytes long. Here it is sanitized:

qemu-system-x86_64 -enable-kvm -name hostname.domain -m 4096 -smp 2 -pidfile /var/run/ganeti/kvm-hypervisor/pid/hostname.domain-device virtio-balloon -daemonize -D /var/log/ganeti/kvm/hostname.domain.com.log -machine pc-i440fx-bionic -monitor unix:/var/run/ganeti/kvm-hypervisor/ctrl/hostname.domain.com.monitor,server,nowait -serial unix:/var/run/ganeti/kvm-hypervisor/ctrl/hostname.domain.com.serial,server,nowait -usb -usbdevicetablet -vnc 127.0.0.1:5156 -chroot /var/run/ganeti/kvm-hypervisor/chroot/hostname.domain.com -uuid 3d9ea1c9-33ce-4e06-81d3-a97005b89e4b -netdev type=tap,id=nic-19f31fbd-f5de-4b57,fd=9 -device virtio-net-pci,bus=pci.0,id=nic-19f31fbd-f5de-4b57,addr=0xd,netdev=nic-19f31fbd-f5de-4b57,mac=aa:00:00:XX:XX:XX -netdev type=tap,id=nic-0e7f0c3a-e19b-4ae2,fd=12 -device virtio-net-pci,bus=pci.0,id=nic-0e7f0c3a-e19b-4ae2,addr=0xe,netdev=nic-0e7f0c3a-e19b-4ae2,mac=aa:00:00:XX:XX:XX -qmp unix:/var/run/ganeti/kvm-hypervisor/ctrl/hostname.domain.com.qmp,server,nowait -qmp unix:/var/run/ganeti/kvm-hypervisor/ctrl/hostname.domain.com.kvmd,server,nowait -bootc -device virtio-blk-pci,bus=pci.0,id=disk-066d688d-9f5a-4ca5,addr=0xc,drive=disk-066d688d-9f5a-4ca5 -drivefile=/var/run/ganeti/instance-disks/hostname.domain.com:0,format=raw,if=none,id=disk-066d688d-9f5a-4ca5 -S -runas kvm37

qemu on its own is indeed faster to iterate and learn with.

In production you probably want the host-vm and vm-vm confinement provided by svirt https://libvirt.org/drvqemu.html#driver-security-architectur...

One thing is missing comparing to virtualbox, is the bridge-mode especially when host uses wireless which is very common these days.

virtualbox hacked bridge-mode and made it working well all the time, with libvirt/qemu/kvm you will need write a lot of scripts and click around to set this up.

This is the sole reason I had to stick with virtualbox these years.

I'm a big VirtualBox fan as well and I'm sad to see that Apple M1 (ARM) support isn't really a priority.

I get the impression Oracle is killing VirtualBox through neglect.

Then you need to compare to the (for non-commercial use free ) vmware workstation (player) as well.

- Simple OpenGL accelerated guest graphics (alternatives: intel gvt-g, card passthrough, virgl in qemu (not generally available), SPICE+qxl in qemu (not nearly as performant))

- bridged networking "just works"

- open-vm-tools enable quite an interesting array of comfortable guest integration

- disk image format can be handled by qemu-img for conversion purposes, so it's easy to migrate vmware <> virtualbox <> qemu+kvm.

The graphics part is why for desktop VMs, I prefer the vmware solution at the moment.

This is the main virtualbox feature I sorely missed running kvm. Is there any easy way to configure bridge mode on kvm? I didn't dig too much because I also run docker and kubernetes on the same machine and I don't want to break their network config accidentally.

If anyone knows how to setup the bridge with wireless in libvirt/qemu/kvm following the steps with/without ebtables listed in Debian and Arch wiki, please do tell

1. https://wiki.archlinux.org/index.php/QEMU#Network_sharing_be...

2. https://wiki.archlinux.org/index.php/Network_bridge#Wireless...

3. https://wiki.debian.org/BridgeNetworkConnections#Bridging_wi...

For 99% of the people that want "bridged mode" ipvlan should work with minimal config needed. For the other 1% it gets more hairy.

One annoyance I had with the defaults are the short dhcp lease time configured for dnsmasq, causing repeated dhcp lease logging in the logs. Turns out you can configure the lease time using virsh net-edit. The libvrit docs contain additional useful configuration options.

Another small thing but big quality of life improvement is exporting LIBVIRT_DEFAULT_URI='qemu:///system', so that everything you execute uses the correct connection.

Cockpit is a nice web UI for libvirt:

https://cockpit-project.org/

On debian add backports then it's as simple as "apt-get install cockpit-machines" then point your browser to http://localhost:9090

Do I guess correctly that you have cockpit running on the host and manage guests via the cockpit-machines plugin? If so, do you have any hints on where to find documentation about that? A cursory glance at the repo didn't produce much information.

How annoying that this page uses a non-standard port for the WebSocket "livereload" feature (1313).

It makes my firewall pop up constantly.

Browse with JS disabled by default. Then in uBlock Origin you can temporarily whitelist the page and browse it with JS enabled if it really requires it.

I've been struggling to run Windows VMs on Linux for a while. Are any of these tools a good option for that? I've installed a Windows VM more times than I can count and it hangs after 20-30 minutes of waiting. If anyone has a suggestion for an all in one tool that can help with this, I'm all ears.

cockpit worked great!

libvirt has too many gotchas to be as easy to use as virtualbox, vmware or proxmox. Proxmox also uses qemu+kvm, but is opinionated enough to pick the most appropriate option by default.

Example? For Networking, you probably want the virtio driver for maximum throughput if it is supported on the guest (even Windows has drivers now), using the emulated realtek device/e1000 is very slow. That means you should probably also use the virtio block devices, right? Nope, wrong, work on that was abandoned years ago, use SATA!

The point about storage doesn't seem to be true: https://www.qemu.org/2021/01/19/virtio-blk-scsi-configuratio...

However, I am not an expert and would love to learn more. Intuitively though I wouldn't expect emulated SATA to be better than paravirt of any kind.

Qubes OS (Libvirt/QEMU/Xen) would also be good

I've been quite happy with LXD's support for VMs. Its really easy to pull an image an have a running VM with a single command.

LXD does virtual machines now? I though it does only "containers" (running processes namespaces, like Docker).

Wow, so LXD added VM support! But it seems like still experimental. [1]

[1]: https://lxd.readthedocs.io/en/latest/virtual-machines/

Also useful:

- using Vagrant with the Libvirt plugin. Then there's vagrant-mutate for converting Virtualbox boxes to KVM images.

- VM snapshots with virsh

Proxmox exists.

Reply on news.ycombinator.com