Hacker News Reader

This has to be a distilled model? I don't see any route to 200 tokens/s without serious datacenter hardware

you mean quantized? it’s only got 16B active parameters so it can be pretty fast at decode.

sorry, yes, quantised