nvidia gates it's virtualized GPU offerings behind their enterprise cards, so we're left with ineffective command translation.
IMO: I can tolerate just about every other type of VM overhead, but choppy/unresponsive GUIs have a surprisingly bad ergonomic effect (and somehow leak into the performance of everything else).
If we could get that fixed, at least amongst Linux-on-Linux virtualization, I think virtualizing everything would be a much more tenable option.