Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU | Hacker News Reader

Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU

github.com

60 pointsjoshcsimmons24 comments

To put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175 Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.

That video seems to show it takes ~4 seconds to process 43 input tokens. Considering the amount of tokens coding-agents chew through, even when just exploring and not even implementing anything, feels far away from "acceptably fast". Say one file is 500 tokens (small file), it'd take ~a minute just to process it?

You mention "See later in the reply for the prefill at ~300 t/s" but I didn't see that in the video nor in any of the Twitter replies, could you link where that is supposed to be happening? Is that warm context perchance?

Something to keep in mind with aggressive quantisation, is that it's been empirically proven [1] that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance.

Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.

[1] "Reliability Scaling Laws for Quantized Large Language Models", TUM, https://arxiv.org/pdf/2607.10855

I imagine that it's just the non-linearity of audio makes it easily apparent how much quantization really matters, and that benchmaxxing is so ingrained in LLMs that if you simply run them through benchmarks, they can do well under quantization, when in reality, they have a similar drop off that they do in audio, it just doesn't slam you in the face at every turn as obviously.

I think your leading statement is a bit more authoritative than what the linked paper actually demonstrates. It demonstrates your claimed qualities for specific algorithms, methodologies, and models.

It also explicitly says that Quantization-Aware Training is out of scope, may reveal different trends and has potential for extreme quantization. Heavily-quantized model releases today generally use QAT or something similar, so I don't think the paper holds.

I wish people would clean up the ai spammed emoji from the github readme. The readability is just horrible.

At which quantization? That's the only thing I want to know.

EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.

Everything lower than 4/5 is really bad

Well it's bad compared to the original model, but is it bad compared to what else you could run in 64 GB?

probably; 32-64 is the range where local models get extraordinarily useful, and in my experience any sub-4 quants get ridiculously incoherent, which is why those models tend to try to 'wow' people with tok/s rather than benchmarks.

haven't tried it yet though.

I’m wondering myself: Do bigger models suffer less from quantization? Are there papers?

They seem to degrade a lot less.

You don't need papers, good quants are usually published with benchmark data, for example: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-... which maintains parity with FP8 even at 2 bit.

Always check the label and if pain persists, see your doctor!

The ISTA accuracy looks good on that page, but it's only a chosen handful. Instead, look at most of the graphs on Unsloth's Qwen 3.8 Flash Next - https://unsloth.ai/docs/models/qwen3.8-next

As always, check the quant vs KLD graphs (half way down that page).. I found that not all quants are the same - Unsloth's GLM Q4 -> Q3 degrades a lot faster than their Minimax M3.1 Q4 -> Q3 ggufs.

you can get a lot done with models well under that - the google and qwen 20-30B class models are fantastic

This but also the kv quantization shouldn’t be lower than 8 bits.

Next frontier is achieving good output with that kind of quant dimension

This 100%

Reddit is flooded with these breathless headlines “xyz tokens/s on my light switch”. It’s always quantizations below 4 bits per weight. Mostly useless.

I’m confused too and the slop readme doesn’t help at all.

> Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB

And later

> Standard IQ2_XXS (76.87 GiB, Unpatched)

Except unsloth’s IQ2_XXS GGUF is 91GB, even their smallest IQ1_S is 82.5GB. So I wonder which model/quant the number even refers to.

Also the mentions of “compression” and “MoE pruning” also make it sound lobotomized even more than unsloth’s IQ1 quants.

I very recently spent a lot of time using deepseek v4 flash 0731 before it was superseded by newer more capable things (glm 5.3-flash, v4.1 flash 1017, etc) and I am exceedingly skeptical of anyone that claims to get good QUALITY results out of it at that size. Whatever they're doing to shrink it will make it worse than an unsloth Q2XS quantization of it or equivalent. That size of model needs something like 150GB of RAM to run properly with full unquantized kv cache and not with a ton of training data discarded.

I remain exceedingly skeptical of people who are claiming to run fairly large models in ridiculously tiny sizes. For comparison a 'regular' version of v4 flash 0731 has file sizes more like this: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

Even the 97GB Q2XS will show very significant divergence from the full precision thing. Running a lobotomized model just to say you're doing it is really not accomplishing anything special.

Anyone else finding these AI generated pages really hard to read?

This is quite an obnoxious Readme.md with AI slop. It wraps around llama.cpp but without seeing what platform assumptions it makes its unclear for other readers whether this is worth testing.

It is 2 bit? anybody tested this? is any good?

yeah 2 bit quantization, so probably not that great for actual work

Reply on news.ycombinator.com