HN
Hacker News
Top
New
Best
Ask
Show
Jobs
Comment by tgtweak | Hacker News Reader
Full thread
tgtweak
·
Citing full precision benchmarks then just going with 4 bit quants
View on HN
IsTom
·
It's natively Q4.
hellohello2
·
I though 4 bit (mixed) quant is pretty standard, at least for local models that basically what everyone runs? Unless I'm mistaken it can really be a negligible difference in perf.
Reply on news.ycombinator.com