kamranjon

It's useful logs which i think is an important distinction.

I don't really think the distinction here is relevant. If the end result is the equivalent of lying, cheating or stealing - then the problem still exists and it needs to be solved.

New coding harness that seems to have some novel concepts and one of the pretty cool things on their landing page for it here: https://deepseek.com/harness/en/ is the Every Run is Traceable view:

"Everything the model sees is recorded in an append-only session log: system prompts, reasoning, tool calls and results, subagent scheduling, and every context injection. In the Trajectory view, you can inspect these records by source. Resume, fork, search, and replay all operate on the same event stream."

Seems pretty helpful - have sort of wanted something similar (I use Pi).

They also released this research paper that backs their whole plugin composability system that seems pretty cool: https://github.com/cordiverse/paper

this seems to be a countdown for the 2.4t model - which is gigantic and is not as exciting for those running models locally

I actually have been exploring this very thing!

I think the best option right now, since Apple has raised prices and Mac minis are basically impossible to get your hands on, is to build your own micro-itx machine. I actually built a mini-itx machine, but it does restrict your options a bit.

The Arc series Intel GPUs are what I think make this possible. I built a machine with an Arc b50 - it runs Gemma 26b a4b qat at around 30tok/s with their MTP head and prompt processing sits at around 500 tok/s. The really beautiful thing about this setup is the entire energy envelope of this machine sits at 120w at full load - when idle, it's at 40w and i've done some work in ubuntu to basically intelligently hibernate, which drops it to 0 watts when not in use. You can use a raspberry pi and Wake on Lan to wake the machine up for a overall draw of around 5 watts when not in use.

All in all this machine cost me 1.4k to build - but if you used micro-itx instead of mini-itx parts you could do it for under 1k - it has just 16gb of ddr5 but you don't really need more if you use models that can fit in vram.

I think it's pretty incredible that you can run an actually useful coding agent on a machine with a power envelope that is less than an incandescent light bulb. If you go up to micro-itx you can do even large cards like an intel b60 with 24gb or a b70 with 32gb and run even more powerful models. For all of these intel GPU's you'll want to compile the latest llama.cpp version with SYCL support - they are getting speedups every day, so worth staying on the edge.

You might look at this and and be a bit disappointed by the performance against qwen and gemma models - but this is an entirely open source training pipeline, this is quite impressive and I don't think another model this performant exists with fully open source data and recipes alongside the weights.

I honestly think of the T5 model family to sort of be the real beginning of this open model craze - I know BERT was already popular for classification etc, but T5 was the first sort of generally useful model, was exceptionally simple to fine-tune, and is still in use today (t5 base is still averaging over a million downloads a month on huggingface), has tons of variants and sort of kickstarted this whole community. US labs get a lot of flack but Google has been super supportive and open in a lot of ways that has pushed this whole endeavor forward, even if I feel like they've sort of declined in transparency in recent years with their open models.

1 to 200 is a pretty big spread to between losing 18k dollars and making 3.6 milllion - do you have any actual numbers on the value produced from this 18k investment?

This is pretty interesting, I've never heard of this approach before - do you know if there is a research paper that covers how this was achieved?

The sort of sad but interesting thing here is that while this claims to be a more direct translation, Claude models were certainly trained on all of the pre-existing translations and this is likely going to directly influence the output in the ways you've outlined here. I'm not sure any modern LLM could honestly be claimed to produce a translation that doesn't draw on these pre-existing sources.

This wasn't written with AI... obviously... I feel like there might need to be a new definition for whatever this paranoia is called because it's getting a bit out of hand.

Because street photography is very spontaneous it’s pretty common practice to set an aperture of 8 and just snap away - it’s a helpful trick for rangefinder cameras that often take some time to pull focus.

“Current approaches to low precision primarily focus on converting models trained in full precision to lower bitwidths. We view this as fundamentally the wrong approach…”

Very excited to see how it performs, I’ve been a bit skeptical of the efficacy of converting existing models - really cool to see one trained from scratch in the ternary format.

“While technically true the hallucination rates on modern models is low…”

Isn’t this entirely context dependent? Where did you get the information that modern models have low hallucination rates? I’d love to see the benchmark if there is one, it seems like it would be useful to track.

This seems really interesting - I was curious about this line from the website.

“The whole post-training stack in one CLI. Soup doctors your data pre-flight, picks the method, writes the config, derives evals from your own data, gates every save, and self-corrects reward hacking mid-run instead of just halting.”

How does soup auto tune the hyper parameters and make some of these more complex training decisions?

Do you have an example? Would love to read one.

I love this analogy and think it possibly also applies to search which just doesn’t work anymore and is full of AI generated garbage.

What the hell happened to stack overflow? I don’t think I’ve gotten a google result for stack overflow in nearly 2 years. How have they just forgot how to make competent search?

I didn’t think it’d happen so quickly but I honestly get better results from DuckDuckGo at this point.

In what way is it similar?

This is amazing and I think will probably end up being a pretty important development.

I was just reading this great breakdown of how diffusion Gemma works: https://newsletter.maartengrootendorst.com/p/a-visual-guide-...

In reference to the difficulties with applying this to autoregressive LLMs - I wonder if these type of hybrids might be a good candidate for this approach.

I actually run it as a server - so most of the time I don't have to listen to it right next to me - it's just sitting in another room in my house - but I often am traveling with it and will have it sitting right next to my coding laptop and yea the fan runs non-stop - it's not obnoxious so i can pretty easily tune it out - also airpods/noise canceling headphones help!

The really interesting thing about this is how big of a jump was achieved with just extra fine-tuning here. No structural changes to the model, just more data, compute and time. It makes me pretty excited for the future of small models - DS v4 flash is a relatively small model when compared to the class it's competing with, so likely similar gains can be made applying quality data/training pipeline to other smaller models.

Generally get 20-25tps - prefill is pretty good around 400-450tps. I have been using compaction at around 100k tokens but mostly just cause it was the default in pi coding agent - might see if i can expand it a bit.

Can't wait for the DwarfStar quants - I have been using DeepSeek v4 flash (preview) as my main coding agent for months now (running on my 128gb mbp) - it seems this model outperforms GLM 5.2 on nearly every metric. Thanks for sharing the news, I was refreshing huggingface but gave up thinking it likely would take some more time.

I know many companies are spending quite a bit of money, I don’t know if it bears out that the increased spend has resulted in increased profits, even if there has been some increase in productivity. I think this is the tough situation many orgs are facing right now, drastic adoption without material economic gains.

I haven't spun up thinkingcap yet but I'm aware of it and am intending to try it out soon. How did you find it?

Would you happen to have a link to that interview? Sounds like an interesting read.

Sorry I should have clarified - I meant that a ternary 27b model would outperform a non-quantized or 8 bit quantized 9 or 12b model - which it is generally close to (or much smaller than) in size. So yeah the comparison I was trying to make was between models of equivalent size or models that could run on similarly sized hardware.

I’ve actually been really impressed with the 27b model they recently released - amazing performance approaching 40 tok/s on m4 max and I didn’t run into any quality issues in the small set of tasks I tried. Haven’t gone full coding with it yet but suspect it’s better than say a 9b or 12b model.

Unfortunately Fermion Research appears to entirely AI generate all of their content here, even for the research section: https://www.fermionresearch.com/research/neutrino-8b/

"Neutrino-1 8B was trained natively in its shipping format. There is no full-precision product model that was rounded afterward: the ternary representation is the medium the weights learned in, and the training methods that hold this quality at this depth are the lab’s unpublished work. The findings below are the part that travels."

This statement seems misleading at best.

Both the model page and the release page are basically unintelligible - I don't have a ton of faith in the work here, at least PrismML write coherent releases for their models.

Edit: Another beautiful piece of prose here, I almost wonder if they used the 8b model to generate the content for this release...

"Across the 6.95B coded weights, 62.63% sit at zero and the remainder splits 18.68% plus to 18.69% minus: sign-balanced to a hundredth of a point with no constraint asking for it."

There's a really interesting trend of labs using "proprietary" methods to convert existing models to a compressed ternary format.

PrismML actually targeted the same Qwen 8b model and got it down to 1.75gb here: https://prismml.com/news/ternary-bonsai

I wonder how proprietary it all is though, since the BitNet b1.58 paper has been out for a couple years now: https://arxiv.org/abs/2402.17764

From the wikipedia on 1.58 bit llms: "BitNet derives its performance from being trained natively in 1.58 bit instead of being quantized from a full-precision model after training. Still, training is an expensive process, and it would be desirable to be able to somehow convert an existing model to 1.58 bits. In 2024, HuggingFace reported a way to gradually ramp up the 1.58-bit quantization in fine-tuning an existing model down to 1.58 bits."

The section from huggingface is here: https://huggingface.co/blog/1_58_llm_extreme_quantization#fi...

I just wonder how many of these labs are basically following the huggingface recipe here and possibly tweaking it and releasing models without huge training costs.