The problem here is memory, so while it should be possible to run inference on a 4090, you would have to constantly move data from VRAM to RAM, though I don't have benchmarks to prove it I think it would be slower than 10x
does that mean it could at least run? even if a magnitude or two slower?
Coz it uses all of the memory so the moment you'd have to spill it to RAM or worse, disk it would be not x4 but x100+ slower
Sure but that would get you to one inferences around every second. I wait around 1-2 minutes with some online tools for 4 inferences. So seems like it might work fine for a local no data leak version?
VRAM, you need enough. That’s sort of a binary thing: either you have enough or you don’t. 1x4090 does not.
But at 4x another GPU, they also don't have the vram, right thus 4...?