colingauvin

What do you consider a local model? I'm running DeepSeek Flash version 4.1 and it's outstanding.

It actually benchmarks quite well. Honestly not practically worse than the Q4 native weights.

You could probably run a Q2-Q3 with 2x DGX Sparks. So...$12k at today's prices. Plenty of folks are running DS 4.1 Flash with Strata and some GPU + bulk in system memory. Maybe more like $3k-$4k? But speeds come down quite a bit.

I have absolutely no words. If this is real, I can't even imagine the future anymore.

For API, this seems competitive based on the released benchmarks. For running at home, this is pointless when DS4.1 is more or less equivalent but half the total parameters and less than half the active parameters.

It's definitely not a solved problem, and it has been kicking my ass. I don't just want metadata tagging/categorization, I want something that identifies the networks between the information, graduates topics to actual objects, and routinely consolidates/synthesizes narratives that can be easily looked up.

I initially tried annotating everything three times, then judging the annotation with the same model. That failed predictably, so yeah just logprobs right now, judged on a hand-annotated dataset. Mostly correlated - ~80%.

Yes, once it's in LocalAI, I intend to try it out.

Seeing the fine-tuning guides that are coming out with LFM3, I am now considering it :)

Would be a fun experiment.

One thing I have been using LFM 2.6 for is annotating emails, text messages, and other metadata. I keep it all ingesting into one giant database and annotate it all as it comes in on an RTX A4500. The LFM annotations are basically triage, and then slowly annotations trickle into my halo model based on lowest confidence (right now that's DeepSeek 4.1 Flash 1017) for an upgraded annotation. (Agents can then search this database for context).

Excited to roll out LFM 3 to replace it, looks like a big step up. Right now LFM 2.6 gets trustworthy annotations on just over 40% of data points. Just waiting for these to get into LocalAI.

No weekends. We need you to get back to work on the next 8B.

What? We know exactly what it takes to run: 2.8 TB VRAM for FP8...1.4 TB VRAM for FP4.

Of course we'd have no idea if Claude or ChatGPT were doing this because we can't see their stream of thought.

Good article.

>I do think there is a real but moderate Jevons effect here: because inference is so cheap now, I’m much more likely to implement marginal automations than I would have been before. But it isn’t unbounded. In practice, it’s maybe 2x or 3x as much code as I would have wanted at the old price, because I still have to specify what to build, review the output, and maintain it. When the price per task drops 20x, and my usage only goes up 2x to 3x, total dollar revenue into the ecosystem (and net GPU demand per user) still collapses by 70% to 85%.

I don't think this checks out. Jevon's paradox doesn't say when electricity becomes 20x cheaper, someone already using electricity will use 20x more on the same thing they already do, or even 20x more at all. It means a combination of more people and more enterprises will use it, and use more of it.

Concretely - at Claude API prices I'm not going to have AI run a bunch of things that I can do without it. But at DeepSeek API prices, I might.

Fundamentally, I think that we aren't even imagining a number of the use cases for AI, in large part because they don't exist yet. But I do think they will eventually exist in the next 20+ years. Right now we are strapping AI to everything, not reimagining life with coherent artificial intelligence. Similarly I couldn't have predicted something like Walmart+ or Amazon Prime back in 1995.

I'm running it at 3 b.p.w. and it really feels better than Flash 4 at 4 b.p.w. for me. Some of these EXL3 calibrated quants are like magic.

It also has 196B ngram params that sit on disk. So it's really pretty comparable to GLM 5.3 in size.

I was running DS731 until Qwen 3.8 Flash Next (so what, like a couple weeks? The pace has been torrid). I switched to Qwen for image input, then GLM 5.3 Flash. At the time, Qwen felt like a minor step down from DS731, GLM 5.3 Flash feels like a major step up from Qwen, so I'd say slightly better than DS731. I realize that's not exactly what the benchmarks say, but DS731 definitely flailed around on some of the more challenging things that GLM mostly gets for me.

I got this frequently with on GLM 5.3 Flash with GLM's actual plan, I do not get this with self hosting.

Pretty much just this, but tweaked to have I think 64 active conversations be cacheable, which drops the context pool to ~2M.

https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Spark...

I also had to downclock my Sparks (Asus) slightly, to 2000 Mhz. Otherwise they'd overheat and crash.

Yeah, MiMo Pro is the first Chinese model I can really clearly identify as an upgrade over GLM 5.3 Flash. K3, Qwen 3.8 Pro, GLM 5.3, DS4.1 Flash all feel around equivalent to me. Oddly lots of people have derided MiMo but I have found it able to tackle slightly tougher problems. For instance, I tried GLM 5.3 Flash, DS4.1 Flash, and MiMo Pro on making a new map layer for my Garmin watch, and MiMo was the only one that found the actual best method (.img file for OSM).

It is unreal how good GLM 5.3 Flash is. There's an ~4 bpw EXL3 quant that fits on 2x DGX Sparks. I'm not sure it's worth it now ($7k/Spark) or even when I bought them ($4k/Spark) but there's something about knowing that the whole internet could go down and I could still have this model running at 50-60 tokens/s with 1M context, indefinitely. It's just so good. TensorFold hits 100+ tokens/s in aggregate with 2-3 M total context pool. It's like having Claude at home.

There's a few comments (boldly) criticizing the ruling for being technically incorrect. If no child was abused, how can it be a child abuse crime?

I would counter with this: If it is technically impossible to tell the difference between AI generated images, and real ones, how could anyone ever be prosecuted for having presumptive real ones in their possession ever again? Everyone tried would simply say "AI" and that'd be that, unless there was a victim to testify, or it was a case on generation of it etc.

So AI images of this nature do have victims, just slightly less directly.

Another adjacent question: How would you feel if it was an AI generated image of a specific child. What is the precise threshold of victimhood that is necessary?

You can jail the actual pedophile in that scenario, you cannot jail an agent. The real question is whether the person who instructs the computer to do the thing that leads to hacking should be liable, or whether the company that made the tool that when used unsafely lead to hacking should be liable. Gun manufacturer's typically are shielded from liability. They also have a lot more disclaimers on their products though.

Technically beta keratin is helical. That said you are correct, there is no requirement to be helical for fiber diffraction. It just turns out it's very difficult to have non helical uniaxial symmetry in nature.

When I was studying spindle shaped viruses by electron microscopy we'd get beautiful fiber diffraction patterns in the fourier transform of the micrographs. I always wanted to try and solve the helical parameters but never got around to it.

Specifically the antiparallel base matching was the important bit, agreed, but minor nitpick. The first alpha helix wasn't solved until 1951, so one year earlier. Their ubiquity was very novel.

The author's configuration is..... Interesting.

I don't disagree with the title. This spring folks were saying "I use Qwen 3.6 27B (or 35 A3B) as a daily driver". They either had low standards or were lying. The models were good enough technically, they capable enough, that you could see the potential, but the coherence was really not great compared to frontier. They'd lose track of what they were doing, mix things up, etc. Technical ability definitely arrived before coherence.

Then DeepSeek 4 Flash came out and suddenly it was like... Ok. You can run a local model that keeps coherence, and is decent technically. Then we got a rash of models that are both good technically and coherent: DS4 Flash 0731, Qwen 3.8 Flash Next, Qwen 3.8 27B, GLM 5.3 Flash, Hy4, MiMo 2.6 Flash... It just has no stopped. We have definitely crossed the threshold from interesting to very usable.

I like the solar bit. Not sure why the author is using a 4 model setup. My tests with Qwen 3.8 have convinced me it's completely fine general purpose, and dedicated small/vision/etc are useless. I can get 110 TPS single stream on 2x Sparks, with 5M total context, concurrent I think some folks are hitting 200-250 TPS. There's no reason to use anything else right now, it's really remarkable. For an Opus 4.6/4.7 quality model.

I am eagerly awaiting the release of Qwen 4 Flash.

I certainly don't pay $25 to register. Maybe it's different for businesses? But for me on a 2024 truck it's $300-$400/year in Montana.

Not my project but wanted to share. My 2x Spark config with Qwen 3.8 Flash went from 37 tps to ~90 tps (prose). It's running phenomenally well and really impressive.

Not comparable. The markets can still charge wherever they want, but it needs to be priced per item and not per customer. In your house counterexample, you could set whatever list price you want.

Thanks for sharing. Very interesting.

Could you share more about this?