More cache never hurts. I'd imagine there were GPU perf gaps, though they were hard to distinguish from CPU-based performance hits. The most notable issues were random latency spikes caused by the multiple layers of hypervisors, which interfered with some games and occasionally caused audio/video desync on Youtube.

I ultimately tore down that setup and just swapped to dual-boot. The steps needed to set up high-performance VFIO (i.e. clearing enough contiguous RAM for 1GB Hugepages) meant most of the benefits of VFIO never really materialized for me.

Yeah I'm very curious as to how this effected 99% framerates and frame pacing.

I suspect only a modest hit to average framerate, but I can only imagine it hurt the actual max frametimes which make it "feel choppy" even if the framerate is still higher then your monitor's refresh rate.