Does FP4 support really matter if local inferencing is RAM bandwidth limited anyways?
I think we should put more effort into compressing the models well to be able to use local GPUs better.
I think we should put more effort into compressing the models well to be able to use local GPUs better.
FP4 only helps batched inferencing.