Xen vs. KVM: network, CPU, and disk performance compared

Those graphs are brutal. The graphs are 3d and on angles, so numbers that are the same actually look different. Just a complete mess. I know thats not what the post is about, but It makes the post considerably less useful and the graphs considerably less usable.

We use virtual machines to emulate larger networks, particularly when we need true independent actors at the edges. We chose KVM over Xen because it is easier to get an embedded system to run on emulation than it is to run on a hypervisor kernel. Also, some systems are just not able to be hacked at in that way. Our performance has been good, its nice to see how our intuition shakes out compared to actual measurements.

Some notes I have on top of the article:

Network performance monitoring:

--iperf has a tendency to use a ton of cpu. This can impact network performance metrics. Lately I like using the 0mq performance tests, slightly modified to last longer than the default 10s.

--In my setups KVM adds approx .5ms latency to network transit, when using para-virtualization through a tap device and bridge, having the host os do the routing.

--The virtualized e1000 works better in most cases than any other NIC KVM will virtualize.

CPU performance:

--The best results for KVM come when setting cpu affinity, if you have a workload that coincides nicely to 1 vm/core, there is a performance boost.

I found it odd and jarring that for the first 2 graphs, KVM is blue and Xen is green, while for the CPU graph, Xen is blue and KVM is green. For all subsequent 3-way comparisons, KVM is back to blue and Xen is back to green.

A bit off topic, but is anyone using BSD jails these days?

With the way the Xen kernel has lagged, this in very encouraging. For KVM, all that I have to do is enable it in my latest kernel sources.