More cache never hurts. I'd imagine there were GPU perf gaps, though they were hard to distinguish from CPU-based performance hits. The most notable issues were random latency spikes caused by the multiple layers of hypervisors, which interfered with some games and occasionally caused audio/video desync on Youtube.

I ultimately tore down that setup and just swapped to dual-boot. The steps needed to set up high-performance VFIO (i.e. clearing enough contiguous RAM for 1GB Hugepages) meant most of the benefits of VFIO never really materialized for me.