maciejgryka

Does anyone know what the license for this model is? Specifically any word on restrictions about what it can be used for?

That’s like asking whether fuel consumption was productive or leisure as the number of cars on the road increased. It’s both! I don’t think you can separate one from the other in any reasonable way.

Thank you fur writing this nichochar.

I have many similar thoughts, but one thing that worries me is also trying too hard to do the optimal thing. I often find that my own expectations collide with reality (kids’ moods, random events necessitating change of plans etc). So it’s important to switch from “rationally making shit happen” mode during work time to “vibe and see what happens” during family time. This mental switch is hard.

Ha that’s a really interesting comparison! I guess the difference is that OR directly sends revenue to model providers, so maybe they’re more likely to continue working together? But I can totally see it go the other way once a provider feels confident enough their users won’t switch away. At that point the OpenRouter tax, however small, will be a problem to solve.

I’d bet there are as many stories of businesses failing because of inability to ship quickly as there are about focusing too much on your tooling instead of delivering value to customers. Both failures are dangerous! And depending on your market, product, team etc. a different spot on that spectrum is appropriate.

In some places shipping quickly is part of the value. In others, the product works and delivers value and having slow releases (implying eg a manual release process) is a feature.

What this article advocates for is certainly a valid lens, by IMO it’d be a mistake to take it as universal.

We benchmarked which small language models are most tunable and which deliver best performance after fine-tuning. Tested 12 models (Qwen, Llama, Gemma, Granite, SmolLM) on 8 tasks.

TL;DR Qwen3 family is the best overall choice, small Llamas improve the most after fine-tuning.

Huh works fine for me, even when not logged in to Github. Can you try again?

We've been experimenting with small models for structured tool calling tasks and just released gitara. Both the 3B and 1B models turn natural language instructions into valid git commands and run fully locally through Ollama.

Highlights:

1/ Same accuracy as a 120B GPT-OSS 2/ Runs on your laptop using Ollama 3/ <1.5 seconds response time

Examples: - “undo last commit but keep changes” → `git reset --soft HEAD~1` - “what is in the latest stash, show diff” → `git stash show --patch` - “show 8 commits as a graph” → `git log -n 8 --graph`

The repo includes the code, tool schema, examples, and evaluation method.

I think it’s going to be a while before we see small models (defined roughly as “runnable on reasonable consumer hardware”) do a good job at general coding tasks. It’s a very broad area! You can do some specific tasks reasonably well (eg I distilled a toy git helper you can run locally here https://github.com/distil-labs/gitara), but “coding” is such a big thing that you really need a lot of knowledge to do it well.

I think this is a description of how things are today, but not an inherent property of how the models are built. Over the last year or so the trend seems to be moving from “more data” to “better data”. And I think in most narrow domains (which, to be clear, general coding agent is not!) it’s possible to train a smaller, specialized model reaching the performance of a much larger generic model.

Disclaimer: this is pretty much the thesis of a company I work for, distillabs.ai but other people say similar things e.g. https://research.nvidia.com/labs/lpr/slm-agents/

I’d encourage everyone, who finds this appealing to check out how Ecto works in Elixir. It’s all functional & immutable goodies and pipelines are built into the language and idiomatic.

Definitely looks weird at first glance (and Ecto is kinda weird even if you’re familiar with Elixir), but it’s such a joy to use once you grok it.

Highest-end fidget spinner I’ve ever seen. Instantly appealing to my inner 6-year-old.

Yes, compute is absolutely the limiting factor today. Not only because the space of hyperparameters is huge and having more compute would make it easier/possible to explore. But also, weirdly, because inference becomes increasingly important for training, which means even more compute! A lot of work these days goes into getting better data and it turns out that using an existing large model to create/curate data for you works really well.

> “Retrieval-Augmented Generation” is nothing more than a fancy way of saying “including helpful information in your LLM prompt.”

There's hype and FOMO for sure and you're right that there's lots to learn from information retrieval work. But why be dismissive of the whole thing? People learning from past research and applying it in new contexts seems like a good thing and there's people (like https://x.com/bclavie and https://x.com/jobergum), who are legit experts in information retrieval and are showing how to apply it to RAG properly.

For sure, it's only worth doing if you actually have so much relevant data that it doesn't fit in the context! This is definitely the case for us for this problem, but it's not universal.

This is one of the things we learned recently about building production workflows with LLMs. Happy to answer any questions/feedback here <3

I have no actual info on this, but I always assumed they'd compute some mutlimodal embeddings of the screenshots to then retrieve semantically-relevant ones by text? And yeah, they'd have to do it using on-device models, which doesn't seem out of reach?

I recorded myself trying to read through and understand the high-level of this if anyone's interested in following along: https://maciej.gryka.net/papers-in-public/#scaling-monoseman...

Isn't "two models slapped together" basically how all of these things work, starting with CLIP? Not sure about GPT4o, obviously, I don't think they released any underlying architecture details?

I haven’t tried this yet, excited to see how it can do segmentation by outputting series of coordinates! That's something I just assumed transformers will generally be bad at.

I hear you and am planning to try DSPy because it seems attractive, but I'm also hearing people with a lot of experience being cautions about this https://x.com/HamelHusain/status/1777131374803402769 so I wouldn't make this a high-conviction bet.

Nice, looking forward to seeing that! Someone else pointed me towards https://github.com/DAGWorks-Inc/burr/ which also seems related in case you're curious.

Ha good point, I did say "let's have another shot" when I just meant another try at generating! FWIW "few shot prompting" is how most people refer to this technique, I think (e.g. see https://www.promptingguide.ai/techniques/fewshot), I haven't heard "priming" before, though it does convey the right thing.

And the reason we don't really do it is context length. Our contexts are long and complex and there are so many subtleties that I'm worried about either saturating the context window or just not covering enough ground to matter.

I'm a little surprised to hear this, my experience has been a little better. Are you using GPT4? I know 3.5 is significantly more challenged/challenging with things like this. It's still possible to make it do the right thing, but much more careful prompting is required.

Oh this is fun! So you basically define personalities by picking well-known people that are probably represented in the training data and ask them (their LLM-imagined doppelganger) to vote?

I think instructor is great! And most of our Python code is typed too :)

My point is just that you should care a lot about preserving optionality at the start because you're likely to have to significantly change things as you learn. In my experience going a bit cowboy at the start is worth it so you're less hesitant to rework everything when needed - as long as you have the discipline to clean things up later, when things settle.

Yep, it's a DAG, though that only occurred to me after we built this so we didn't model it that way at first. It can be the same LLM with different prompts or totally different models, I think there's no rule and it depends on what you're doing + what your benchmarks tell you.

We're running it in prod btw, though don't have any code to share.

Yeah, the fact that different models might react differently to such tricks makes it hard. We're experimenting with Claude right now and I'm really hoping something like https://github.com/stanfordnlp/dspy can help here.

I wouldn't be surprised to see it help, along with the "you'll get $200 if you answer this right" trick and a bunch of others :) They're definitely worth trying.