V100 is a beast if you feed it the right format. A lot of these older cards don’t do well with lower quants. So instead of using Q4, try Q8 or FP16.

If you’re still getting poor performance, point an Astra agent at it and ask it to benchmark and find the pressure points.

Also speculative decoding has changed the game completely. I’ve managed to 2-3x’d the theoretical max for my setup. Apple Silicon is a bit more limited here, and cannot deal with large DFlash block sizes. But AMD and Intel chips are game.

Good luck and godspeed!