HNHacker News
TopNewBestAskShowJobs

Ir0n

6 karma · joined October 15, 2024

submissionscomments
Ir0n··on Running Qwen3.8 at 255 tok/s on a single 5090
we achieved this on stock nvfp4 model which is within 5% of fp16 version

we then apply lossless techniques only

dflash2 as spec drafter and patched ninfer inference engine (down projections for short token lengths and adding tool call related schema to drafter shortlist)

see: https://github.com/meshapplied/goethe/pull/4

benchmark numbers are measured using see bench verified traces at avg input 28k and concurrency 1