Hacker News Reader

Comment by a11r | Hacker News Reader

I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality. I'm running 4-bit quants on an RTX Pro 6000 rented for approximately $1/hour and getting about 1.2 million tokens out and 40 million tokens in per hour with caching. The quality of 4-bit quant is good enough for difficult but well-scoped coding tasks. Here is the inference stack I am using: https://www.reddit.com/r/BlackwellPerformance/s/FrKwk3GoDK

View on HN

I just ran a set of benchmarks, ninfer-3090-qwen3.8-27b (so mix of Q4 and Q5) vs strata-qwen3.8-flash-next-iq3_xxs (so Q3):

│------------------- │ Ninfer-3090 │ Strata

│ Code generation │ 52/78 (66.7%) │ 70/78 (89.7%)

│ Code completion │ 40/50 (80.0%) │ 44/50 (88.0%)

│ Total------------- │ 92/128 (71.9%) │ 114/128 (89.1%)

│ API failures------ │ 10 │ 5

- Ninfer generation: ~122 min total.

- Strata generation: ~142 min total.

So strata is a little slower, but keep in mind that ninfer-3090 is very optimized for a Qwen 3.8. Standard Qwen 3.8 runs at 20 t/s, this modified version can do 50 t/s (but it's extremely long in it's thinking, it just goes on and on.

This is on a 3090 that will crash unless power capped, with a Zen 2 CPU, 64GB DDR4 with a PCIe that refuses to go higher than 8x (basically pretty crappy all in all).

Yet with some tweaking and optimizing I still manage to get strata to run at 40 to 60 t/s.

That strata has been optimized on my Oh My Pi conversations. So when I'm using it, it's probably faster and closer to ninfer in speed than during those unoptimized benchmark tests.

Anecdotally, I've found that 27B at 4bit hallucinates a lot more than QFN at 3_xxs, particularly for less technical reasoning.

I've tried to use both for online comparison shopping. QFN not only seemed less delusional but also made useful observations and problem solved ways around many different website access issues.

You can use 4 spaces on HN to get code/monospaced formatting:

    │------------------- │ Ninfer-3090    │ Strata    
    │ Code generation    │ 52/78  (66.7%) │ 70/78   (89.7%)
    │ Code completion    │ 40/50  (80.0%) │ 44/50   (88.0%)
    │ Total------------- │ 92/128 (71.9%) │ 114/128 (89.1%)
    │ API failures------ │ 10             │ 5

| Benchmark │ Ninfer-3090 │ Strata |

| --------- │ ----------- │ ------ |

│ Code generation │ 52/78 (66.7%) │ 70/78 (89.7%) |

│ Code completion │ 40/50 (80.0%) │ 44/50 (88.0%) |

│ Total │ 92/128 (71.9%) │ 114/128 (89.1%) |

│ API failures │ 10 │ 5 |

I wish Markdown tables worked here.

They don't but you can still format things in an actually readable way and be nice to people.

People also often us _italics wrong_, just coz it works that way elsewhere. Works totally fine with a "*" instead on here. Just go with the program and you can make beautiful posts even on here. Heck you have AI now so you don't even have to remember anything any more. There's even a `help` link next to the input box and just to write this I double checked it and I was even misremembering and using two too many spaces for monospace! I really do wish `these would actually work` but hey, you really can't have everything!

Yes, why did somebody somewhere decide underscores should be italic? They're most clearly underline, as in the days of dial-up BBSes. (-:

In my private C coding benchmark, Strata's Qwen Flash Next Coder IQ1_M beats Luna low and runs at 125 tok/s on my 5090

Not as fast for me. I was able to snatch a spark before the horrible price hike, and I have it at 40/50tok/sec. Before I had the great 3090 setup which is what led me to get the spark. That 3090 is now relegated to creating videos of my kids doing dumb things.

I now backordered two more sparks, before the hike... hoping they will honor the agreed price and dont cancel on me. They should arrive in a month,

Its evidently clear the frontier labs won't keep on giving us cheap inference for much longer. I also watch in disbelief how people say a sub is cheaper, which probably is. But you loose on so many other things (privacy, predictability, censorship...)

I pray for the flood of CXMT high speed memory and Huawei accelerator chips

> it's extremely long in it's thinking, it just goes on and on

Known issue with this model, I recommend setting thinking to 'medium' instead of default 'xhigh'.

Ill check for this in my setup. I thought it came with only one thinking level causing the “meandering” as it refers to its own long CoT.

I’m doubting the quality of the model, even comparing with luna 5.6, it is giving such bracingly wrong answers. It claims other tools have changed the code when pressed why it made a mistake. That code or tool didn’t come close to working on the project. It is scary.

You can also set a thinking budget in terms of how many tokens it can spend inside the reasoning/thinking phase.

Unrelated... just use your just locally installed qwen 3.8 and instruct it to figure out why you arent at 16x.

For me, I didnt know I was on 4x and I was able to, giving it enough permissions, with a good harness like PI, to figure things out and go 4x -> 8x -> 16x.

It probably will take some opening the case and switching ssds around, but it's amazing how powerful these models have become.

For the 1 dollar per hour, may I ask (without trying to incite things) why you would not just buy a ChatGPT Plus subscription or other 20 dollar subscriptions? Do you like the privacy?

maybe ethics as well. maybe people like sleeping well at night.

Those plans are super slow right now. Nowhere near 100t/s.

I think Opus 5.5 is now fast enough with the inference stack improvements that they did. Sonnet 5.5 is around 140 tokens/s when I had measured it last time. A $20 subs on claude and using only sonnet 5.5 will last a lot

I was firmly wanting to move towards local llm, and since the Sonnet 5.5 and recent Opus speed increases, I'm putting that on hold until local inference speeds can be improved. Not sure how much juice there is to squeeze there, but I'm hopeful.

While this may be true, their throughput is practically unbounded. this has been a primary challenge to using local models at work, where its normal for me to have 1-6 sessions churning away at once. So while 100t/s on a single session is very fast, theres no way you could get a cumulative ~500t/s from a consumer setup like this, do the bandwitdth bottleneck on a single card/cpu/ram/ssd (tends to be gpu ram bandwidth is the limiter in my setup, but they all have respective limiters to this kind of throughput)

I’ve had to use almost all resets that OpenAI allotted me. But I agree. I can get probably 1000t/s overall from them if I’m using subagents a lot.

Does it matter if those 100 tokens per second are way crappier?

Access to models is currently being priced below it's actual value. Largely because the market hasn't settled yet. Once it does the prices to access AI will increase -signifigantly.

It will be less of an issue for people who have already got the equipment and the practice running locally than it will be for people who have simply been relying on ChatGPT.

Prices on openrouter are not subsidized and are very affordable.

It's pretty interesting to see people still parroting talking points from ~2023 when there is plenty of data to the contrary.

People think claude costs $200/month, it actually costs $200/month and all your labor to train their models, improve their product, and all your best ideas and work that they will feed back into their company. Its very expensive

You can opt out of model training with a checkbox, though I doubt most companies respect this completely. At the very least, I imagine they find loopholes to exploit.

But on the other hand I'm giving them my best and worst ideas and the value I get in return outweighs what I'm contributing to them. I also don't have the problems of initial HW cost and maintenance.

> You can opt out of model training with a checkbox

Do you trust that a company who can't even do basic IT ops monitoring are capable of respecting a "no training checkbox"?

"Whoops my bad, you are right, your data may have been in the training pool. They are now forever in weights that'll be reused to train every future model. Too bad you can't do anything about it."

Why do you care? It's their legal/compliance problem, not yours, if it turns out they didn't (though I don't understand the urge to deny them this in the first place, it's not like it hurts you or deprives you of anything).

Great, then they will PII-scrub my sessions before feeding them into the training pool :-)

Yer I also use service with zero data retention promises but it is hard to really trust them for me... They trained on basically all of the existing material, legal or illegal already and are starved for true human input. They are also in a heavy arms race were basically only a handful of companies will survive.

Yes. Training is necessary to make an LLM. They don't grow on trees, you know.

Their own TOS will still include your "anonymized" data in their "research", with no explanation as to what that means. You need an enterprise account to get real zero data retention.

Yes, and? Why would I care? Why would I begrudge them? I'll get the improved product, too, that will benefit from not just my data, but everyone else's to.

Don't get the greed and the Dog in the Manger mentality so ever-present around AI detractors.

Did the mathematicians who got scooped on Navier-stokes benefit from the improved product or did their lifes work get stolen?

>I'm a little skeptical of going below 4-bit quants due to the potential for significant degradation in quality

There's literature on this actually, if you want to get that low the model really should have quantization as a pretraining target otherwise larger models that are quantized after collapse.

In fact, to do this effectively, you need somewhere near 50x chinchilla to do QAT on super tiny targets like 1-bit or even 2-bit. These large models already need an astronomical amount of training data that doing proper QAT that small isn't really feasible unless there are breakthroughs in the architecture.

4-bit models and smaller _can_ perform well but not through just naively quantizing the existing weights of a large model.

> 4-bit models and smaller _can_ perform well but not through just naively quantizing the existing weights of a large model.

Doesn’t that describe the majority of quants on higgingface?

Yeah, there's a tradeoff. pre-training with QAT needs more data and a ton of hardware, so it's easier to just take an existing open weight model and quantize it. This works, but it'll underperform a model that had quantization as a training target.

Matters more for 1-bit and 2-bit models.

My we should crowd fund distillation with 2 bit quants on a deepseek 4.1 flash and see how it does ? Bet it does well ..

Can you say more about where you're renting the RTX Pro 6000 for $1/hour?

vast.ai?

I am renting spot VMs from Nebius. I've tried a variety of other providers like Vast and Spheron. Vast worked well for renting 5090s but I like the large memory and pricing I'm getting at Nebius for RTX Pro 6000. The extra RAM really matters because I need PLE to offload the ngram to RAM.

Just as curiosity, how long does it take to set the environment up and running?

Is it viable to start/stop it multiple times per day?

Yes, you can probably get the whole thing up and running in about an hour the first time. If you pause and restart, it takes about 15 minutes to load the models from disk into GPU memory, so budget for cold startup time.

How does it take fifteen minutes to read <100 GB into GPU memory? Shouldn't that be limited by SSD speed with everything slower than a minute being a terrible ssd?

A lot of cloud platforms have terrible slow network storage. They also might need to compile the GPU kernels fresh as they might not have a persistent CUDA cache.

It also depends on the runtime, vllm is unbelievably slow at model loading compared to llama.cpp

this is not right. vllm can easily load safetensors model at >5GB/s if not faster when setup right. did you make the compile cache persist? if you use docker, you should bind mount the kernel compile cache, so they don't need to be recompiled each time vllm restart.

I just had claude code set it up for me and create some startup scripts. The longest part was the model download.

I have to concur with you here because I'm running 3.8-flash-next in the largest unsloth Q8 version on a 256GB system for daily use, and I've recently compared it head to head vs a Q4 that I have on a 128GB system and the Q4 is just plain not as good. The Q4 is given only more basic tasks.

I can't even imaging letting a Q2 anywhere near a code base that I care about. To generate a bird on a bicycle as a test? Sure. Actual things I don't want it to screw up? nooooooooo.

There is a magical quant GSQ-RCO that at 3-bit XS beats the full unquant. I have tested it. It is on par with the full unquant on DeepSWE. Give it a shot, it is magical. ITSA lab, they basically devised a way to selectively quantize parameters. I am a GSQ-RCO truther at this point... it is so fast on my V100s and I can't find an instance where it has acted weird or fell short of the full quant on agentic coding. I Have it 3d model in blender, all sorts of stuff. It's just... good.

ISTA-DASLab's IQ3_S quant is really good https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RC...

What are you running it on? I'm getting mixed results using the IQ3_XXS quant which supposedly matches baseline, it feels significantly degraded.

I've run DeepSeek V4 Flash on DS4 on single DGX and standard model weights on aDGX cluster. There was some degradation going to the hybrid 2 but quant but really not much.

The larger the model, the more you can go down. K3 in Q1 will match and likely beat Qwen3.8-Flash-Next.

$1/hr sounds great. Where are you getting it for those prices?

[flagged]

No? Per someone else's comment, presumably they're using vast.ai which does indeed list the 6000 for $1/hr

Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Apologies. I had recently read (I think from someone on here) that they were in some discord where those sellers advertise. Whoever it was mentioned that the service was incredible value for what they got, but admittedly it was for toy projects, there was no written agreements, and uptime was something like 98%. I should have found the source and linked, and will try to do better overall in my replies!

That's cheap, too. Which hosting service are you using?

$1/hour ? I need that deal as well.

Not the original commenter, but Upcloud has them at that price for spot instances: https://upcloud.com/pricing/

Reply on news.ycombinator.com