Hacker News Reader

Comment by hyperpl | Hacker News Reader

I use qemu and never really understood the need for any further abstraction such as libvirt over it's cli usage. Can anyone tell me what there is to be gained other than needing further setup, packages and systemd services?

View on HN

This is a blog post I wrote about the KVM userspace architecture, it goes into why Red Hat uses Libvirt in its virtualization products.

https://www.redhat.com/en/blog/all-you-need-know-about-kvm-u... (fixed now)

Getting a 404

https://www.redhat.com/en/blog/all-you-need-know-about-kvm-u...

It's been probably a decade since I've used libvirt, but I can say why I like using ganeti:

- I can get a list of the host machines in my cluster and how much memory and storage they have available.

- I can easily move VMs between hosts if I want to evacuate a host for hardware/software/firmware maintenance.

- It has the ability to set up DRBD backed VMs and live migrate between the host nodes.

- List what machines are running and on what hosts.

- Start and stop commmands don't require me to remember the settings on individual VMs, the qemu commands that get run are something like 700 characters long.

> - Start and stop commmands don't require me to remember the settings on individual VMs, the qemu commands that get run are something like 700 characters long.

Shouldn't be the case. I put my VM args into a script, but it's only like 20 lines?

Actually, looks like I under-estimated by half. One of the random VMs I run has a command-line 1361 bytes long. Here it is sanitized:

qemu-system-x86_64 -enable-kvm -name hostname.domain -m 4096 -smp 2 -pidfile /var/run/ganeti/kvm-hypervisor/pid/hostname.domain-device virtio-balloon -daemonize -D /var/log/ganeti/kvm/hostname.domain.com.log -machine pc-i440fx-bionic -monitor unix:/var/run/ganeti/kvm-hypervisor/ctrl/hostname.domain.com.monitor,server,nowait -serial unix:/var/run/ganeti/kvm-hypervisor/ctrl/hostname.domain.com.serial,server,nowait -usb -usbdevicetablet -vnc 127.0.0.1:5156 -chroot /var/run/ganeti/kvm-hypervisor/chroot/hostname.domain.com -uuid 3d9ea1c9-33ce-4e06-81d3-a97005b89e4b -netdev type=tap,id=nic-19f31fbd-f5de-4b57,fd=9 -device virtio-net-pci,bus=pci.0,id=nic-19f31fbd-f5de-4b57,addr=0xd,netdev=nic-19f31fbd-f5de-4b57,mac=aa:00:00:XX:XX:XX -netdev type=tap,id=nic-0e7f0c3a-e19b-4ae2,fd=12 -device virtio-net-pci,bus=pci.0,id=nic-0e7f0c3a-e19b-4ae2,addr=0xe,netdev=nic-0e7f0c3a-e19b-4ae2,mac=aa:00:00:XX:XX:XX -qmp unix:/var/run/ganeti/kvm-hypervisor/ctrl/hostname.domain.com.qmp,server,nowait -qmp unix:/var/run/ganeti/kvm-hypervisor/ctrl/hostname.domain.com.kvmd,server,nowait -bootc -device virtio-blk-pci,bus=pci.0,id=disk-066d688d-9f5a-4ca5,addr=0xc,drive=disk-066d688d-9f5a-4ca5 -drivefile=/var/run/ganeti/instance-disks/hostname.domain.com:0,format=raw,if=none,id=disk-066d688d-9f5a-4ca5 -S -runas kvm37

I'm like you and use the qemu command line. But libvirt does make it easier to define cpu groups, thread affinities and the like and without needing to be root to do so. Something like this. There are other ways to accomplish this I think without libvirt, but it gets a little hairy.

https://www.reddit.com/r/VFIO/comments/9iaj7x/smt_not_suppor...

I am not an expert but libvirt is not only for qemu but it supports various virtualization technologies such as Xen, LXC, VMware. So if you decide to stick with QEMU then I guess you don't need to use libvirt.

libvirt is mainly a consistency wrapper. So you can use kvm or lxc.

As for what it “gives”, I suppose I’ve never migrated a VM outside of libvirt, though I’m not 100% sure if that’s not possible with plain old qemu+kvm

When I started using virt-manager I started there because I was specifically looking to replace Virtual box due to having kernel modules for VB sometimes block kernel upgrades. So I guess I was looking for a GUI since I was on a desktop anyways. (Just have some dev VMs so I don't have to clutter my desktop with running MySQL etc for some projects and so I can match the distro of the production server)

So I guess, it worked, didn't look into raw qemu. I also use it infrequently enough that I would have to relearn the CLI all the time.

It's not just about abstraction. VirtIO is significantly more efficient than SATA emulation, and as it is built into the Linux kernel it just works. There is also a Windows driver package that adds VirtIO support, but it's a bit tricky to get it to work when porting in an existing Windows VM. VirtIO also makes it possible to do USB relay.

VirtIO is very usable via QEMU, without libvirt (naturally, because in the configuration described in the article, libvirt is just calling QEMU). It is usually as simple as `qemu-system-x86_64 -drive file=/path/to/my/disk,if=virtio`.

There's more cool stuff coming in this area too. For a long time there's been the virtio family of protocols for shuttling IO to something outside QEMU to handle. Originally that was always KVM and the implementation is called vhost. Then later it became clear that these same messages could be sent to another user space process to handle instead (called vhost-user). These work great for creating virtio devices in the guest. But operating systems like Windows don't have virtio device drivers in-box, so it's a little annoying.

Recently, a new protocol to replace virtio has been defined. It is modeled on vfio ioctls and currently only can forward to another user space process, so we're calling it vfio-user. With this protocol, it's possible to emulate any PCI device rather than only virtio devices. Projects like SPDK (what I work on) can now use this to present fully emulated NVMe devices into guests and back them with whatever actual storage is available (a file, something over the network, a real NVMe SSD, etc). This allows an OS, including Windows, to boot from the virtual disk using it's in-box NVMe driver. This hasn't quite made it into a QEMU release yet, but it's close!

qemu on its own is indeed faster to iterate and learn with.

In production you probably want the host-vm and vm-vm confinement provided by svirt https://libvirt.org/drvqemu.html#driver-security-architectur...

Reply on news.ycombinator.com