firelex

I think the v1.2 idea of building a better zero-shot model at 0.8B is a dead end. v1.3 switches to the adapter route. That's what I would focus on - same performance and much better accuracy in your domain. And because adapters are cheap to train, take no memory and you can switch them extremely fast, they cost next to nothing.

Correct. We trained - but did not release - Arcade adapters for Jeff based on (1) a generic text representation of 25 or so Arcade games and (2) video frames. While the adapters played some games extremely well (e.g., Pac-Man), they sucked at others. Game play often involves strategic decision-making, which a classifier, particularly a 0.8B one, just is never going to be very good at. The applications lie elsewhere.

In my view, the most exciting thing about system 1 models is that we can finally innovate outside the frontier labs again. As for Jeff, we trained v1.2 on a single RTX Pro 6000. Granted, for v1.3 and Jeff-Code, we had access to a DGX H100 and a DGX B200 (we had to, to run through all the benchmarks), but that's still orders of magnitude less than the smallest AI lab. So it democratizes AI development, and that, I think everyeone will agree, is a good thing.

Jev follows a very different philosophy. It wants to be your zero-shot partner everywhere. Jeff and similar applications - particularly Jeff v1.3 - doesn't try to do this anymore. It puts the value in the adapter. And the Jeff adapters will likely always perform better than Jev zero-shot. It's just a matter of specialization. Horses for courses.

The new Jeff v1.3 adapters cover 15 applications. I think more research is needed on where to draw the line between fine-tuned base models and adapters. Depending on where that line is drawn, different applications become possibel.

I had the same thought: What were we all missing these past three years? And I think it's just that while everyone was staring at LLMs, the hardware has become a lot better and we now have 0.8B models that are extremely well-trained, so that allows us to look at the "old tech", such as classifiers, in a new light. It's also interesting because it decentralizes innovation: You don't need billions of hardware to do interesting experiments in this space.

For anyone interested, we tried the adapter approach in a coding agent harness: https://jeffhub.ai/notes/jeff-v1-3

What's interesting is that these are two adapters: Jeff making simple decisions (which tool to call, how to parameterize it) and Jeff routing between different version of Qwen. There are lots of applications for this approach.

In terms of Jeff-Code specifically, treat it as an experiment. Swift-Qwen is faster, but also - I understand - worse on benchmarks. The interesting bit is the concept here.

At GTC2025, NVIDIA more or less pressed a Founders Edition DGX Spark on me (and many other conference attendees hoping for a future of local models). Full disclosure: I'm the guy letting out a screm when Jensen announced the Spark at the conference. But like everyone else, I was disappointed when we found out what the memory throughput was. I didn't use it for a year, but then Qwen3.8-Flash-Next came out, which I think is the first model where the Spark's architecture actually mattered, and the paperweight became useful. I even bought a second one in preparation for Qwen 4-Flash, and so far, I'm happy with it.

The RTX PRO 6000 was a present to myself a year ago. Same story: the drivers didn't work at first, so for a while I used it to heat my flat. Now they work, and it trains everything here without a cloud bill.

So yes, renting is the saner choice, and nothing here needs local hardware. The final training runs were about 24 GPU-hours on one RTX PRO 6000: 2 hours for the 0.8B base, 3.5 for the 2B, and 18 for all nine adapters. Verda lists that card at $2/hour on demand and $1 on spot, so roughly $50, or $25 if you're patient. The real cost was everything around it: days of synthetic data generation on the Sparks, the runs that didn't make the cut, and a 27B benchmark that kept a Mac busy and warm.

If you just want an adapter for your own task, one rented GPU for an afternoon is plenty. The adapter kit is in the repo.

Hi HN. Jeff is a set of small, open-weight Qwen3.5 and Gemma fine-tunes for zero-shot classification, with respectable out-of-the-box performance, meant to be slotted right into code (or fine-tuned further as needed). You give them a situation and a list of options; they return a calibrated probability for each, in one forward pass, with no text generation. The 2B scores 83.1% on a five-benchmark panel (Jev's published figure: 83.0%); the 0.8B decides in about 28 ms on an M4 Max. Apache 2.0, with a Jev-compatible API (I'm not affiliated with TypeSafe).

When TypeSafe released Jev a couple of weeks ago and then AutoJev appeared, I wanted to see if I could replicate the experiment using only small language models on local hardware. So everything ran at home: one RTX PRO 6000 for training, two DGX Sparks running Qwen3.8-Flash-Next to write the synthetic data, a MacBook for testing, all monitored from my phone over Tailscale.

The caveat: the published Jev and AutoJev numbers are on a different sample of the same benchmarks, and Jeff's overall score comes from classification-style tasks (96% on Financial PhraseBank, 86-89% on RAGTruth, both above Jev). On multi-step reasoning it's behind: BBH 64-68% against Jev's 94%, and about 50% on JevBench's hard tier against 73%. That isn't surprising, and I don't think it matters: no 0.8B or 2B model reasons like a large one, and nobody should expect it to. These are extremely fast judgement-callers. In one of my apps I use the 0.8B for voice navigation; a quick fine-tune (about half an hour on one GPU) took it from 32% to 96% on held-out commands, at about 40 ms per decision.

The fun part: games, as a zero-shot test. Games aren't the ideal zero-shot test, but they're fun, and TypeSafe did it with Jev too. There was no game data in training. Each turn the code describes the situation and the moves in words, and the model picks one; the options say what each move leads to, never which one is right. Over 20 episodes each:

- Doom (ViZDoom): Jeff 0.8B 6.55 kills per episode, the same as a hand-coded bot and as Jev's published run. Jev's prompt spells out the aiming rule and takes about 212 ms per call over its API; Jeff gets "the nearest monster is a little to your left" and decides in about 29 ms on my Mac.

- Frogger: 10.3 crossings, level with the hand-coded bot (10.25), and 10x the untrained base model (1.0).

- Pac-Man: 57 of 98 pellets, about 60% of the bot's score and 2x the untrained model.

Videos of every run are linked in the README.

Lessons learned:

- System 1 models are here to stay. Being able to process unstructured data at software speed inside an app is extremely powerful, and being able to do it locally is fantastic.

- A small model is a classifier, not a planner. Models of 0.8B-2B don't reason like Qwen3.8-27B or Jev, and they don't need to: present the options the right way and you get 40+ decisions per second, depending on your hardware.

- Fine-tune it if needed. If zero-shot isn't good enough for your task, a short fine-tune on your own examples is.

- Wording matters enormously. Giving Frogger's final step the same words as every other forward option ("safe, and one row closer to the goal") took one episode from 15 crossings to 23. Before that, the frog just stayed on the last log.

- Bigger isn't better. The untrained 2B is already more risk-averse than the untrained 0.8B (in Doom it prefers turning away from the nearest monster), and training made it hesitate in Pac-Man. That's probably why the 0.8B beat the 2B.

- Benchmarks don't predict play. Untrained Gemma 4 E2B beats both untrained Qwens on the benchmarks (62.5%) and plays every game worst: right most of the time, but not reliably, and in a real-time loop the mistakes compound.