How would this compare speedwise to Bonsai 2 on the same GPU?
This model has higher retention because of the weight complexity and the quant method we built, and the speed loss is compensated via kernel optimisations and speculative decoding.
might be a good characteristic for a quantized model to have ..