Builder of thingz, lover of old robust tech like unix, prolog, lisp, APL, forth, postgresql, but don't mind the shiny new things ... right now, I'm all in on GenAI, Local LLMs & LLM driven agents, team llama.cpp, building LLM agents since 2023
Local clusters built around old GPUs. I have had cluster of P40 GPUs, MI50s gpu, and now I have a cluster of cmp170hx. I'm always looking for the next cheap GPU dump and then I get as many as I can before it get's popular.
They gained popularity because THEY ARE GOOD, not because they are open weight. There are so many other models that are not popular. Do you know about arcee and their models? every used inkling? how about laguna? these are American open weights, they didn't take off because they are not as good as the rest. Do you know about the command-x series from COhere? probably not, this is a Canadian model, folks into local were really into it 2 years ago when they were keeping. do you know about dbrx? Yeah, databricks an American company also gave us a model. Anyways, my point is the competition is strong and giving away for free doesn't matter if it can't keep up. It's expensive, to send bandwidth, storage, gpu compute to run a model when there are better options.
Their last 2 models were a disaster and didn't live up to their self reported evals. I hope this time around is much better. I'm happy tho that they are still figuring out how to get better even if only measured against themselves. This is great, the more options we have the better for everyone.
With Anthropic going public, I expect a lot of model releases from the competition signalling to the market. "We are still around, please leave some money for us too"
why would they think that? I'm not a pro, but I know you can't serve more than kv cache pool that you have. If you want to run concurrent requests, you need to have more kv cache pool than your context window. The entire thing is weird, my bet is they were thinking most people will be using about 100k cache, compacting, etc and not nearing 600k ever.
Doesn't matter, they could have marketed it as 1 million token window, you would then infer, get to 600k and get an error. You won't be able to finish your work, and you won't get credit for 600k tokens.
"We deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens. We were aware of this mismatch before launch. "
Did they advertise it as 1,048,576 token when they only had room for 681,280 tokens?
Have you heard the cost of health insurance and seeing a doctor? Some people don't see going to hospital as an option. Better for them to have a local model than no options. But let me tell you how I use it for health, I take all my health records, lab results, every thing, I throw it in there for a second opinion, it explains things to me, it suggests questions I should ask on follow up, possible options, etc.
My local model never says no. I never get banned. If I want to do security stuff, reverse engine, there it goes! Do I wanna chat about politics? No problem! Do I want it to analyze medical stuff? I don't get to hear that it's not a doctor and I should call 911. Do I want to explore legal docs, no problem! There are lots of things that cloud models will say no to! I read that don't even agree to help with advanced biology and chemistry topic much. I also never run out of tokens, no 5 hr limit. I can run it 24/7. I don't run out of tokens. I can run 16 sub agents at the same time, non stop! 1-4 billion tokens a month? No problem. Any agent harness I want! No restrictions. Privacy all day. My own true system prompt, not something someone has determined, no filters. Furthermore, there are many open weight cloud models that don't have equivalent online. For a while before cloud models got good at math, the best model at math was DeepSeekv2 Math and there was no API. The only way to use it was to run local. All sorts of audio models, image/video models, etc. No multiple subscription needed. Private UI for my family, kids included (I can guard what kids infer). Got a friend that really needs some tokens? I give em access through private VPN. Telling someone that they are paying for decades of Claude for local AI is like telling someone they are paying for decades of public transport when they buy a car. Sure, you are right, but ...
Why? AI is taking jobs. getty images just filed bankruptcy, stackoverflow went from heavy use to almost no use. AI is taking jobs, tipping toeing around it isn't the best thing. It needs to be said, and then what next needs to be figured out because first we talk about 200k jobs, then we will next talk about 2 million and then 20million
Yes you can, but IMO, the mistake people make when running huge models at slow speed is that they try to put it through an agent loop. Don't, but they are great for one shot output that you can then pick up and clean up manually
Too early to call, I ran it through the weekend and compared with Qwen3.8-Flash-Next, DSv41F-1017 at FP8 and Q3.8FN at Q8. Both were great, but surprisingly, Q3.8FN had the better result. I was modelling and generating simulation code for a complex system. I used both because I had a bunch of images that needed to be used for input to steer the output. I think these models have strengths and it's best to experiment and find what works of you. I didn't find MiMo2.6-Flash in the same tier, but MiMo2.6-Pro is a beast.