Can you share the source for the parameter count (125B A6B)? I didn't see it anywhere in the page.
3,052 karma · joined April 10, 2013
It is incremental, the more a pattern appears in the context, the more likely it was to continue appearing in future turns.
So the model was likely trained to end a reasoning trace with a single linefeed and a `</think>`. It was also likely trained that two consecutive linefeeds sometimes produce an "Actually..." sequence.
So you can think of it as:
- the first time the reasoning trace was parsed, the two trailing linefeeds followed by </think> were added to the context by the template.
- next time the model was finishing a reasoning block, it added an extra linefeed instead of just closing directly with </think>. This slightly increases the chance that the next token will begin an "actually" sequence instead of closing.
- If it caused an actually, that was added to the context, further increasing the chance of a self correction at the end of the thinking block. The more self correction paragraphs are added, the higher the chance that following turns will have more.
- Eventually it can result in a state where it enters that loop forever (or at least for a very long time).
> Is the “autoparser” vulnerable to injections somehow?
The autoparser was (and still is) incorrectly parsing a trailing linefeed as part of a reasoning block. The were two ways to fix this, both of which must be implemented for the fix to be complete IMO:
- fix the autoparser definition to ensure remove surrounding whitespace is not returned as part of the text blocks
- trim leading/trailing whitespace in the encoding phase, so it fixes bugs or even "injections" where the client deliberately adds the whitespace to trigger problems.
For this specific issue, the maintainer later fixed by trimming the extra linefeed before passing to the template.
> How do you distinguish between user text, model text, and metadata, or is there ambiguity in the parsing?
That is model specific. Ultimately, a token stream is being produced and parsed by the inference engine, and each model uses different tokens/formats. The goal of the autoparser engine was to simplify the creation of parsers for new models by inferring the delimiter tokens from the chat template.
The llama.cpp API server returns pre-parsed data, so clients don't need to do any parsing to know what is a thinking block, a text block or a tool call.
Parsing/encoding is one example: A couple of months ago I've debugged a reasoning loop bug in Step 3.7 Flash on llama.cpp that was caused by the parser capturing an extra `\n` as part of a reasoning block. It was something that only manifested at longer multi-turn agentic sessions, and the extra linefeed was steering the model into making reasoning self corrections that only got worse with longer sessions (more details about this issue: https://github.com/ggml-org/llama.cpp/issues/24181#issuecomm...)
No inference engine is perfect, but I feel that llama.cpp is the most reliable way to run language models locally.
That is a lot, what is your laptop hardware?
One issue I have with Gemma is that they seem to use old architectures that rely on full attention, requiring a lot of RAM for context and quickly degrading speeds as context is filled.
Qwen 3.5+ is much better in that regard with its super efficient context. Even on Macs, speeds take degrade much more slowly.
Maybe a sign that they didn't have SVG pelicans in the dataset
Codex is really good in my experience, especially due to its native sandboxing. Deepseek seems really well versed in its tools, including update_plan and knowing when to request sandbox escalation.
Is that a bad thing?
I suspect the value is in showing the potential that LLMs have in developing new breakthroughs.
One thing that might not be obvious about about DSV4 is how much innovation the Deepseek team implemented in its architecture. When llama.cpp fully supports its lightning indexer, the full 1M context will only require about 6G of RAM. So even though they are similar in size, I believe Deepseek will be much more efficient in that regard.
> I wonder if Hy3 can compete there
Highly depends on how well Hy3 is resilient to quantization. DSV4 is useful even at 2-bit quants.
This Q5_K_M quant should be near lossless and fit with full 256K context in about 100GB of RAM: https://huggingface.co/AesSedai/Qwen3.5-122B-A10B-GGUF
However, I cannot build a good mental model of a software component that I didn't write myself, and that can affect future maintenance if that component is not properly decoupled from the rest of the system.
Very interesting, will be following Zig development more closely. Thanks for sharing!
If the compiler will treat functions that call a library function (Future.await) as special and change how the call site is compiled, why not just have an `await` keyword that when present will convert it into a state machine that can be suspended/resumed?
In other words: What is gained by not having a keyword that changes how a function is emitted if the compiler will change it anyway based on detection of a library call?
I had some trouble understanding how the async/await mechanism works:
var foo_future = io.async(foo, .{args});
defer if (foo_future.cancel(io)) |resource| resource.deinit() else |_| {}
var bar_future = io.async(bar, .{args});
defer if (bar_future.cancel(io)) |resource| resource.deinit() else |_| {}
const foo_result = try foo_future.await(io);
const bar_result = try bar_future.await(io);
The part that I don't understand is what happens when you call future.await(io). Will the IO implementation somehow suspend the current function and resume once the future is resolved? If so, does that mean that every function in zig is a stackless coroutine?
Does that mean it no longer deletes or changes tests to make it pass?
The recent 1 million line vibe coded PR suggests it is not so reliable as a dependency.
It is higher than 110GB. MacOS allows up to 125G of the RAM to be shared with GPU, so it is certainly less than that!
> HF link is broken though!
Doesn't seem broken to me, but you should be able to search for tarruda/Qwen3.5-397B-A17B-GGUF on huggingface.
If by ROI you mean saving more money than using paid APIs, then I don't think it is worth it. All you gain is full sovereignty over your AI usage.
I run 2.54 BPW 397B Qwen 3.5 GGUF on a 128G mac studio at 20 tokens/second generation and 200 tokens/second processing. I'm not suggesting it matches the performance of the full BF16 model, but I did run some benchmarks locally and the results were pretty good:
- MMLU: 87.96%
- GPQA diamond: 86.36%
- IfEval: 91.13%
- GSM8k: 92.57%
So I think we have been at the "frontier capabilities at home" for a few months now.