Hacker News Reader

Comment by incomingpain | Hacker News Reader

That's full quantization. If you run Q4 or Q8 you can run this on <$10,000 equipment.

My experience with heavily quantized models is they do better than a similar sized unquantized model but don't really perform anywhere near the pre-quantized model.

>My experience with heavily quantized models is they do better than a similar sized unquantized model but don't really perform anywhere near the pre-quantized model.

People have tested it. Q8 has essentially no drop in quality, Q4 is measurable but still not realistically a problem. If this impacts you, just pay for the commercial saas option.

This assumes that the benchmarks are representative of real usage scenarios. I'm not saying that there is bad faith, but that benchmarking is really hard in the context of LLMs.

>This assumes that the benchmarks are representative of real usage scenarios. I'm not saying that there is bad faith, but that benchmarking is really hard in the context of LLMs.

It's a fair point, but the conclusion is 'i dont know'

I could assume that it gets better because it'll keep to simpler code.

No point in running anything but full quantization.

Quantization doesn't work? Really?

And if you add in heavy sparsification it should fit and run on a raspberry pi.

So in around 6 months, we will see that the person who bought this H200 in the listing just got scammed for $250k and will realize that you just needed specific quantizations to the model and a few optimizations to run locally.

Unless they want to train their own model, buying this for inference for $250k is unnecessary and still isn't enough for a full production deployment.

It's already sparsified from the 150T parameter model..

It took me several hours to realize that 150 trillion parameters is a reference to the number of synapses in a human brain.

Reply on news.ycombinator.com