liuliu

Great post. I looked into the reasoning trace, and it seems DeepSeek 4.1 Flash 1017 does know all the games you listed in the post, so there is no knowledge gap. Not sure about Qwen though. I will not be surprised if it doesn't know all these games.

These are famous games tested here though, and I tried to feed one problem to DeepSeek 4.1 Flash 1017 and it comes back at 66% confidence it is a Pac-Man (incorrect). That smells to me the problem setup heavily hint towards these 2D games. Might be resolution related to be honest.

On a laptop, yes. That's why it is not a primary device and you can revoke so it won't have continued access to your messages.

On the phone, no. You need to have a few 0-day to break into Secure Enclave to get access to Signal related keys.

If you think a device with storage mechanism that can simply connect to another computer and view all your keys is safe enough for your communication needs, then it is fine, you don't need to use Signal.

People use Signal because they don't want to be compromised when they just leave their phone / laptop unattended for a couple minutes on a bus or a train. They don't want to be in a situation if one day their laptop gets computer virus and they have to assume all these accounts are compromised and have to restart their digital identity all over again.

Might be the first then? I cannot remember. I disliked the last part of the story line, but not sure if that is the season 3 or season 2 + 3.

I enjoyed the first two seasons of Star Trek: Picard. Always knew this might happen some day, but always wished it can keep going.

I haven't evaluated this one, but for what I have, it seem the pplx-decider-v1.2 (and its -flash variant) are pretty good. Anyone evaluated all these open-source decision models on their own tasks? (I evaluated on MonitoringBench).

I think that is true for GLM-5.3 to some extents, this is not true for DeepSeek 4-series models as they compress KV aggressively.

SSD weights streaming. The actual weights is 552B, then engram.

Still need 2 or 3 days to properly quant this. But it is really good model to run if you have 64GiB or larger RAM on Mac. On M5 Ultra, you get ~3400 tok/s for prompt processing and ~50 tok/s for token generation, quite usable state.

The problem with SSD offload is the prefill time suffers, which is not really obvious.

MoE, especially these Chinese ones trained on low-bits already is actually quite quantizable to lower bits, unlike smaller / dense models.

Also: given that the human written bots are almost certainly in the training data, the winning rate against human-written bot seems to be not well-calibrated. You would expect at least they are 50% or better, given that they can "memorize" the best human-written bot.

Worth digging into the protocol and see what went wrong.

FWIW, single M5 Ultra with 256GiB runs DeepSeek 4.1 Flash 0910 just fine (~3200 tok/s prefill, ~45 tok/s). M5 Max w/ 128GiB runs GLM 5.3 Flash also fine.

I also benchmarked both on quality benchmarks (MathArena Apex / GPQA Diamond etc) to make sure the quality is what you get from the official numbers.

My own: https://huggingface.co/drawthingsai/DeepSeek-V4.1-Flash/tree...

DeepSeek 4.1 Flash 0910 is perfect for M5 Ultra 256GiB. Running it fully resident in RAM, prefill at ~2500 tok/s and decode at ~40 tok/s. Probably tons of room to improve from there.

What are you talking about? The model is native NVFP4, why you run it at any precision higher than that?

> Is an open-source vibe-coded creative suite really worth cheering on?

No. They suck the oxygen out of the open-source ecosystem for other more credible software, such as Photopea or Gimp or FreeCAD.

Because that's not possible (to have a GPT 5.6 Sol level model). People won't believe this and will keep dreaming, but intelligence is not free and 8GiB (shared with OS and other processes) is too small to be useful. Whether it is possible for 48GiB or 64GiB (meaning useful for model would be ~16GiB to 24GiB) with external fast storage (SSD), OTOH, is a question mark.

Just add to system prompt: “ask permissions for files and folders you might need upfront. Don’t annoy users with permission request in the middle of the flow.”

It is! The directory grant is once and the sandboxed app can save the bookmark for the future access. As a agent program, you can display this so users can revoke / temporarily disable access for a period of time: https://developer.apple.com/documentation/foundation/nsurl/s...

Agree, I think for CLI harness, there is no good alternative, you either have access permit per terminal app (which is not ideal), or I guess that is what Apple means they will "address" it?

I don’t quite understand why people complains this is bad for AI agents. Local Code (https://releases.drawthings.ai/p/public-beta-of-local-code-b...) doesn’t require full disk access, when you ask the agent to deal with some files it doesn’t have access to, the built-in ‘permit’ tool will trigger the OS folder grant interface and that information will be recorded both by Apple and by the app so it can be revoked later if you want. That allows you to not give full disk access to the app to be useful.

If you are interested in high-end models with high-end Apple Silicon, also try out Local Code: https://releases.drawthings.ai/p/public-beta-of-local-code-b... It is currently in TestFlight (and will open-source next week), supporting vision with DeepSeek 4.1 Flash, Qwen 3.8 27B and DeepSeek 4 Flash 0731 without vision. Custom quants & SSD streaming to make these big models work with 64GiB and above devices (and of course, Qwen works with devices with 16GiB and above).

Both are probably single-token decode performance, which is reasonable to show. Otherwise agree RTX 5090 should shinebetter with NVFP4.

When people benchmark MLX related quant models, they really need to publish numbers on benchmarks. You cannot take this as it is what you get of the original models. MLX uses pretty simple quantization methods so at lower bits without QAT, it is just not as good quality as llama.cpp ones.

Correct. If just stopping criteria, that is less contaminated. The question gets muddier once you also use it to determine hyperparameters during small-scale runs.

When you run benchmarks while training, isn't that the definition of contamination? Asking because I am not sure if this is normal in big labs now.

I sort of understand DSH arch, and their choices, but still baffled by why you would want the agent to change its own agent loop other than the system prompt (in that extension, memory / soul / tools / skills).

Yes, I heard you! I think one of the issue Draw Things inherited is the baggage of supporting too much models. Once you settled on a model, then it is just "try recommended settings", and prompt.

The model part is unfortunate, but luckily converging now.

On the LLM side, hopefully it is not an issue too as long as getting aggressive at pruning models.