Yes, for GPU passthrough you usually assign specific cores to the VM and isolate them (to reduce hiccups caused by other tasks using the core), so in this case you'd just assign the cores with the extra cache.