kamranjon

I follow llama.cpp pretty closely as I use either llama.cpp itself or projects that depend on it all the time, and one thing that I don't think gets talked about is the sheer scale of community involvement. It seems like a logistical nightmare, but somehow thousands of different contributors are opening hundreds of PR's every week and getting them merged in to support various hardware or implement a new pattern or algorithm from a recent research paper. It's really quite awe inspiring for me to see, and think it is in no small part because of the leadership of ggerganov - so I'm happy to see that he is sticking around and plans to keep building this incredibly useful tool that has grown into a huge community at this point.

Did you post in the wrong thread?

I think this account should be banned.

it's funny that the tagline is Radically Open, but you're immediately hit with http login - maybe this was the wrong link?

They said Opus 5 medium - which does have an intelligence score of 59 (you have to select it manually from the dropdown to see it)

Yea I am testing through OpenRouter - have you noticed 3.7 flash being significantly faster?

They've interestingly left out any mention of speed.

I have been testing 3.7 flash against 3.5 flash and it seems to lose every time in overall latency. Every benchmark I've seen seems to suggest the opposite[1] - that 3.7 flash is significantly (at times 2x) faster than 3.5 flash - but I have never been able to prove this out in real world use cases.

Has anyone found their latency numbers to actually be accurate? Is this why they've toned it down in this release? For context, I'm testing larger generation payloads that take 8-10 seconds in 3.5 flash and 15-25 seconds in 3.7 flash. Lowest reasoning settings in both cases.

1: https://artificialanalysis.ai/?speed=intelligence-vs-speed&m...

I am running 3.8 27b at q6 quant with 160k context on a 32gb video card (arc b70 pro) - I quantized the kv cache at q8 - that is the only trick really - works great.

Hmm, makes the 2 bit quants actually seem pretty reasonable…

Trained on roughly 1/9th of the training FLOPS - it’s the pretty incredible that they are making these advances and at the same time sharing their learnings in these papers - I wish we saw more of this from US labs.

I actually do this with my MBP - it's a LLM server when I'm working - and then when I'm not it's just a really great machine for video editing and other media work.

Where did they say that? My understanding of this 3.8-Flash-Next release is that it's a MOE (as per the title of the posting here, 125B a6b)

Have you tried FreeToken yourself? I was hoping to find some benchmarks on their github but took a quick pass at their research paper and it seems they're showing ~2x performance on qwen 3.6 35b when compared to llama.cpp - but llama.cpp is so sprawling and has so many options I find that a difficult comparison.

I think you misread, it’s 170gb/s for base M6 model and 1.2tb/s for M5 ultra.

You would want to get the M5 pro version with 307gb/s if you were interested in running local LLMs.

1tb would likely be ~$20k - given the current >$10k price tag of 256gb. Would you still be considering it at that price?

You can get the m5 pro in the Mac mini with 307GB/s at 64gb of memory it’s $2899

Yeah the article feels as though its describing Khan Academy from 10 or more years ago - it is quite odd.

"What he has never had is pedagogical knowledge: an understanding of how people learn, what motivates them, what makes the difference between someone who pushes through difficulty and someone who types “IDK.”"

It is almost as though the author thinks that Khan Academy is an organization of 1, and that they don't employ learning scientists and have a content team and don't publish research papers in the learning science field.

Is anyone familiar with the laws surrounding police basically operating their own pseudo cell towers?

I would assume this would be highly illegal for individuals, what sort of hoops did law enforcement need to jump through to get this type of approval? Did FCC need to rubber stamp this?

Hi there! I actually thought your Dia models were amazing and very natural sounding, I haven’t tried qwen 3 tts yet - has your focus shifted away from building your Dia models and shifted more towards hosting and infrastructure?

Just wanted to share this, I found it was a really nice resource to understand how diffusion Gemma worked: https://newsletter.maartengrootendorst.com/p/a-visual-guide-...

The really interesting thing to me was that they didn’t need to train this model from scratch they just used their existing MOE checkpoint:

“To convert a decoder-only model (Gemma 4 26B A4B) into a denoiser, we can make use of something it is not directly using when generating tokens, namely the logits of all tokens!”

What makes me hopeful about this release is that possibly this same conversion can be applied to other open models and we might see a bunch of diffusion versions of existing local models. It’s exciting stuff!

I wonder if this could be used by insurance companies to determine premiums?

Are you using the recommended settings for temperature and such? https://unsloth.ai/docs/models/qwen3.8#recommended-settings

Often times I run into issues like this it’s because I am using settings for a different model or just forget to set them up.

I think if a real human doctor completely fabricated a drug usage history for one of their patients, that would also be news.

What size context are you able to squeeze in with less than 2gb of headroom? I have had some luck using a quantized kv cache but i fear that also decreases overall quality.

A no-thinking pelican! I hope to see more, it's surprisingly good for just 2 minutes.

I actually disagree that this doesn’t mean anything. I understand the contention that it’s not measuring the quality of the model in general, but I think it is measuring something useful.

A good example of this is planning hardware projects - a larger 200b plus model like DeepSeek V4 flash will recommend parts like motors, real time clocks, voltage regulators etc and it will do so providing exact model names and specifications.

I wouldn’t expect a smaller model to encode all of this information, but it is helpful to understand where that cutoff is because it changes what the model might be useful for. It is a very crude way of measuring because it comes down to the balance of training data at sizes this small - but I do think it conveys something that is helpful in real world tasks.

Have you thought about running a second tier of the Pelican benchmark where you see which model makes best pelican on lowest or no reasoning settings? I think that'd be pretty interesting and might help highlight which models have a baseline capability - even within those I'd imagine the token usage would vary wildly and might give some indications on verbosity as well.

You can buy two b60s for $1300 right now (650 each) if you want a total of 48gb. Intel recently raised the price on all of their gpu's except the b60 series, so they are currently the best deal per gb I think.

Since Qwen 3.6 27b outperforms Gemma 4 26b in most benchmarks I'm not sure the value - also Gemma 26b is a MOE model whereas this is a dense model, so not typically direct competitors at their sizes - Gemma 4 31b comparison would be interesting though.