How to evaluate performance of LLM inference frameworks

>Aggressively pruning LLMs via quantization can significantly reduce their accuracy and you might be better off using a smaller model in the first place.

Not sure that is correct. Quantization charts suggest its a fairly continous spectrum. i.e. an aggressive quant 13B ends up about same as a no quant 7B:

https://www.researchgate.net/figure/Performance-degradation-...

If you're hitting a memory wall it means you're not scaling. This stuff really doesn't apply to scaled up inference but rather local small batch execution