Could you try with
./target/release/chat --model llama3.2-1b-it-q80.lmrs --show-metrics
To know how many token/s you get? ./target/release/chat --model llama3.2-1b-it-q80.lmrs --show-metrics
To know how many token/s you get? Speed: 26.41 tok/s
Full output: https://gist.github.com/simonw/6f25fca5c664b84fdd4b72b091854...So yeah, I think there is a lot of room for optimization, and the only reason one would use this today is if they want to have a "simple" implementation that doesn't have any C/C++ dependencies for build tooling reasons.