calebkaiser

I'm working on something around this that I'm hopeful I can open source in the next couple of weeks. It's just a research project, so not like a product that can be bought and shipped to prod right away, but I'm excited about it. I think a lot of people are working on this right now, at least in part, with things like memory layers and other approaches to "grounding", or with some of the more recent stuff around certification and governance. But there's sort of a split right now between:

- Groups focused strictly on model capability (i.e. benchmark scores)

- Groups really focused on governance, interpretability, control planes, etc

- A small but excitable minority of weirdos (me included) who think they're the same thing

A lot of the stuff that gets loosely clustered under neurosymbolic methods might be interesting if you're looking to dive in. The sort of pattern you're describing is something that a lot of people have taken stabs at from different directions, or at least have tangentially nipped at while working on related problems.

I think that's a common impression. Interestingly, while there are still a lot of questions marks around what makes someone predisposed to addiction, the few things we do know tend to run counter to the idea that people dealing with addiction are overly sensitive to feelings of reward and pleasure. Or that being less sensitive would make someone more resistant.

One of the most well studied genetic risk factors for addiction vulnerability is the DRD2 gene, which encodes the dopamine receptor D_2. People who are particularly vulnerable to addiction commonly have a deficiency here, meaning getting bored very easily and feeling under stimulated/rewarded by typical things is in part what makes riskier sources of pleasure (i.e. drugs) so appealing.

Disclaimer: Not a medical professional, just like reading about addiction and genetics.

I feel for everyone in this situation. I don't think anyone is wrong to mourn the undeniably massive shift happening to something that has been such a big part of our lives, and I don't think anyone is wrong to be excited. I say this as someone who has worked in ML for coming up on 10 years now and is more towards the latter camp, where I love working with agents, and still spend plenty of time writing code by hand and studying. I don't think it's fair to pretend people are being unreasonable or are just "missing the big picture" for being unhappy.

But I also have found that where you work and what AI usage is like in your company is a pretty big deciding factor. Your comment, which I agree with, made me think about this. I consult a decent amount, and I've worked with companies where the engineering culture has adopted AI in a way I describe as "ugly." I should find a better word. But what I mean is that it's sort of like the attitude is "okay, now we can ship more tickets, so everyone take on 8x the workload". And they do a lot of "best practice" kind of stuff around using coding agents--every new anthropic article gets posted to slack, and there's always a new CI process for using agents--but they're fundamentally just trying to do the same work as before but kinda sorta faster. It's a weirdly miserable experience.

On the other hand, I've worked with companies where the engineering culture almost feels exuberant. I think a consistent difference is that in these companies, it feels like the engineers now have permission to try wildly ambitious things. It's not always optimally productive in the sense that more things get thrown away or end up just not really working out, but overall they move significantly faster and do more interesting work. Commonly, it's things they wanted to do before, but simply weren't realistic with their constraints around capacity. Or it's an idea like "this part of the codebase has always been a performance nightmare, but it's also a beast to untangle. I wonder if I could setup an autoresearch loop and let it run all day, even just to harvest some ideas?"

But yeah, when everyone sort of gives up and/or are forced to move an ungodly number of smaller tickets, it is a miserable time.

I had the same thoughts. I like OpenRouter a lot and still use them, but these numbers make me think more about how much more crowded the space has become. In the last year, you've had startups completely pivot into this niche (Respan and Requesty off the top of my head) who have lots of usage, infra companies like DeepInfra have really leaned in hard to the "serverless interface for open models" positioning, some of the popular harnesses that previously were complimentary to OpenRouter now sell their own gateway/subscription (OpenCode, Nous), and the subscription discount for Claude/Codex is wildly lopsided vs API pricing for solo devs.

Sebastian Raschka has a good introductory piece on what Looped Transformers usually refer to in practice: https://magazine.sebastianraschka.com/p/gpt-6-astra-looped-t...

One of the big points is that looped architectures don't necessarily remove CoT style reasoning. They definitely change it, but the model can still be prompted for this kind of output. It just spends more computation on individual tokens. I think the hope, in that particular scenario, is that the model spends less tokens on things like backtracking.

But there are lots of approaches to looped transformer architectures. Some have some kind dynamic loop (i.e. they might loop until they satisfy some kind of equilibrium, or they might have some kind of routing function that makes different tokens loop differently, or any number of other approaches). Others are simpler "loop this many times" architectures. It's a really active area of research with some pretty deep roots, if you want to dig in! This paper blew up a while ago, and it's a looped transformer architecture (though not like the ones discussed here) that isn't autoregressive, as an example: https://arxiv.org/html/2510.04871v1

If you're genuinely curious, I'd recommend searching around about the history of this persistent urban legend. It's really interesting.

But the short version is that there are lots of people who want to buy any given drug and drug dealers sell them. I don't know exactly what you mean by "WAIT for people to demand the more profitable drugs", but there typically isn't a whole lot of waiting. There's a lot of demand out there. Giving away a bunch of free fentanyl to random people in the hopes that one of them instantly gets addicted would be a pretty wasteful and high risk sales tactic, especially given the number of people that want to buy it already.

The original question was "are you sure an OpenAI subscription isn't heroin?" Different impacts of overuse (death vs slop code) feels like a valid point of difference.

This is a weirdly melodramatic analogy.

If you want to flatten everything to "all things habit forming are definitionally equivalent", you'll get some pretty funky results.

I'm not sure exactly what you're implying but if you're curious, there's a lot of writing about the persistence of this particular urban legend. A lot of it starts with 1920s newspapers painting pictures of dastardly drug peddlers luring in children with free drugs to get them hooked.

Turns out that drug dealers by and large prefer the straightforward approach of selling drugs to the huge market of people who want to take them.

...yes? It doesn't literally kill you if use more tokens than usual?

And we're talking about subscription plans from OpenAI/Anthropic here. Switching to a different model and/or harness if the price changes isn't a "the world has run out of opioids and I'm going to need a detox center" level crisis.

It's weird to me how often this analogy gets used because:

1. In most cases, like this one, the product isn't heroin.

2. Drug dealers don't actually do that.

How does the crowding out work here? I'm not being pedantic, I'm genuinely curious about the dynamics as you see them.

My experience in the publishing world is that there is already an infinite supply of writing. Lots of things serve as filters, some for better and some for worse, but ultimately the quality of the writing (however we define that) vs. the field is part of that filter. There is a competition for our limited little attention spans and finite waking hours, and writing better suited to our tastes tends to win out.

I don't see how that dynamic changes with AI. If all writers have more or less equal access to AI, some writers will use it in ways that result in writing I enjoy more than others. Some won't use it at all and still write work I really like. The majority of people writing with or without AI, just as today, won't write stuff I really want to read.

I don't know if I've ever read a piece of SEO-driven content marketing and thought it was particularly profound or that it made an impact on me. Or that I really had an inherently meaningful connection to the author. I sort of don't care if people generate 100x more of that sort of thing. It makes Google searches less effective, I guess? Or it may make my For You Page on Instagram less human-rich?

I've read some nice poetry though. I usually go looking for it. There are a handful of authors I tend to like consistently. The overall volume of poetry published (it's quite a lot!) doesn't really have an effect on my preference for their work. If there was 1,000,000 times more poetry published because of AI, I don't think it would prevent me from seeking out new poems from writers/publishers/curators I like, nor would I probably really notice.

Ive anecdotally never found OpenCode to have a real axis where I find it "best". But I havent tried it in a bit, mostly because I don't have a real gap in use case coverage between Codex/CC/Pi. Anything more specific in need probably gets a custom harness.

Maybe the Go plan is changing that for others?

LLMs obviously qualify as an example of the bitter lesson. The point I was replying to was using Stockfish as an example of how the bitter lesson was wrong, and I was pointing out that Stockfish-like systems are the core example used in the actual essay

The history of the term is actually really interesting. There has been debate about whether or not its the correct name basically since the advent of computer science departments in the 60s.

I don't think the author is "wrong" about the gap between theoretical discussions computation and the actual design constraints of building usable computers. I don't know if that's really a general refutation of proving new things mathematically though. The word "computer" doesn't appear once in the paper OpenAI released, nor do they make any claims specifically about "computer science": https://github.com/openai/math/blob/main/preprints/An-explic...

It's a novel bit of math. Whether or not it means anything in the context of writing software realistically is an interesting conversation, but it doesn't have much bearing on the substance of the result itself. OpenAI is pretty explicit that they view these as research results in the domain of mathematics.

I don't really know what it means. It seems to sort of entangle Google Search as a product with a lot of constituent parts and "search" as a specific operation over Google's index of sites. Talking about the product, ML has been a core part of Google's search product for a really long time. AI summaries in Google Search distill information from sites discovered from a "normal" Google search. The Search product, which now incorporates AI, and advertising enabled by it, are still far and away Google's biggest revenue drivers.

If we literally mean AI as in people paying to use/deploy AI via Google, that's still a small part of their revenue, far less than advertising revenue. If we mean AI as in "every Google product that uses ML", then I guess that's been their core product for a long time.

This is a weird point because the Bitter Lesson spends a meaningful chunk of its short length talking about chess specifically.

People sort of generally refer to the Bitter Lesson as if its key argument is "Large Transformer models are best", but this is directly from the essay:

"One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning."

The second paragraph of the essay is specifically about chess:

"In computer chess, the methods that defeated the world champion, Kasparov, in 1997, were based on massive, deep search. At the time, this was looked upon with dismay by the majority of computer-chess researchers who had pursued methods that leveraged human understanding of the special structure of chess. When a simpler, search-based approach with special hardware and software proved vastly more effective, these human-knowledge-based chess researchers were not good losers. They said that ``brute force" search may have won this time, but it was not a general strategy, and anyway it was not how people played chess. These researchers wanted methods based on human input to win and were disappointed when they did not."

The "brute force search" he's talking about (Deep Blue) used tree search with alpha-beta pruning on top of a ton of human crafted heuristics and a database of game-specific information it could access. Stockfish has historically been an even more "brute force" approach in that it is much more focused on optimizing search--which is one of "arbitrarily scalable" techniques listed in the Bitter Lesson essay--and it has increasingly replaced handwritten human heuristics with neural networks over the life of its development.

At no point in the Bitter Lesson is there a mention of LLMs driving cars, or a suggestion that large transformer models will be the dominant class of model for every single task. It is specifically reflecting on the fact that approaches that leverage search and learning, combined with large amounts of compute, consistently prove to be most effective. Stockfish would qualify as evidence of that, under his criteria.

That's sort of a loaded question depending on what you mean by "smaller sub expert models inside".

At the most abstract level, you can observe patterns in a model's activations and theorize that particular collections of components contribute to particular kinds of operations. There are many flavors of this intuition. In that sense, you can think of there being "submodels" within a model, insofar as different parts of the network can participate in specialized computations. Some reading that might interest you in this area would include things like mechanistic interpretability.

There are also Mixture of Expert models, which are a dominant architecture for LLMs now, in which you essentially have a bunch of smaller FFNs that get routed over/aggregated together when the LLM performs a forward pass. It would be a mistake to treat any single expert as a better task specific model though. They are trained as part of an ensemble, and they're almost always being combined with other experts in the forward pass, so they're more like (to use a loose analogy) regions of the brain that tend to correspond more towards particular actions than others. You couldn't just extract one and say "This is now our Chess model that performs as well as the larger LLM we took it from". You could potentially use the relevant experts as a basis for training a smaller task-specific model, and there is work around expert pruning and other distillation techniques, but it's not as simple as "This smaller inner model is a complete task-specific model as is". And more importantly, that ensemble of experts *is* the LLM you're comparing it to, at least in major part. It doesn't really make sense to treat the LLM as some foreign model that calls the expert. The experts themselves don't often correlate to easily interpretable task domains, as they've been learned via joint computation with the other experts and broader LLM.

Then there's tool calling. Today LLMs are trained to make tool calls (basically execute functions) for particular tasks. Those tools can be smaller specialized models, or deterministic code, or anything really that can be fired from a host environment.

I'd be curious to see stats on overlap among startups by batch historically. My impression is that this has always been a thing. Based on where the market/technology horizon currently is, there are some buckets that are very "startup shaped". Founders who are iterating on ideas using similar criteria will often end up in the same proximity, especially in an accelerator environment. But it would be extremely interesting to see if this has become more prevalent over time, or at least since the explosion in the AI market.

This is a productive line of research with lots to explore, if you're interested. There are different flavors based on what you're describing that you might like. Maybe you already know all of this, but just in case any other reader is curious :)

A classic paper on "expanding" a neural network (the second part of your distil-then-grow loop) is Net2Net. Their goal is basically to get a smaller model's knowledge transferred into a larger one to bootstrap the larger model training: https://arxiv.org/abs/1511.05641 . Bert2Bert is basically the same idea but language models: https://arxiv.org/abs/2110.07143

There's also stuff on "growing" a pretrained LLM that is somewhat similar to what you're suggesting: https://arxiv.org/abs/2303.00980

And then there's a bunch of stuff on "refreshing" weights in the network as well, sort of like pruning but without reducing capacity.

One of the things to explore are assumptions around how information really compresses down in distillation, and what is really happening when we deem a model saturated. There's often an intuition that when you distill the model down, you're finding a sort of "truer", more essential representation of the functions you're modeling. And so if you distill it down and then expand the weights, you've added additional capacity for it to learn more, since the distilled core now contains the important information from the original larger model. But very often that's not actually how it plays out, and the problem is thornier than it seems on its face. Plateaus in training don't necessarily mean the model has run out of representational capacity. And distillation is not guaranteed to preserve the most valuable information. It's entirely possible that the representations the original model has learned for performing on the tasks it is evaluated against are more generalizable than the compressed representations you've distilled.

Initialization is also tricky when you're expanding the weights like this. Non-linearities make it such that splitting the weights directly like you're describing doesn't preserve the function. Not that this means the model can't recover from further training, but it's just not where you want to start training from if you can avoid it. There's also a strong likelihood that the added parameters are redundant relative to the distilled core, which will make learning more challenging. A lot of work in this space as a result focuses on stuff like symmetry breaking. A cool paper you might find interesting that isn't about LLMs, but has a similar "Progressively give the model more degrees of freedom" flavor is ARC-AGI Without Pretraining: https://iliao2345.github.io/blog_posts/arc_agi_without_pretr...

But there's lots of active research around this general space. You should test out your ideas and share them! I can't recall now exactly but I feel like I've seen MoE stuff that touches on some similar ideas.

If you can thoroughly and accurately review 100 * 400 = 40,000 lines of CUDA kernel code per hour, I know roughly 1,000 people who would love to hire you right now.

I don't use Pi very much for actual writing code. I have some specific use-cases where I do, but I'm admittedly just not a person who can reasonably juggle a lot of tools or a super personalized setup. I mostly just use Codex or CC from the terminal, though I've recently dabbled with some GUIs. Basically, once I find something that works, I'm pretty reticent to spend any time tinkering unless I feel a real need.

But what I do use Pi a lot for is as a base for agents. I much prefer it to using an agent SDK. I find its minimalism and extensibility to be a really nice substrate for new projects.

I could easily imagine someone getting to a really productive personal setup with it as well, for the same reasons as above.

There is a bunch of stuff to tease apart.

In general, training a general purpose classifier is something lots of people have worked on for a long time. Large Transformer models themselves are typically "generalists" already, so structured generation and constrained decoding have given you the ability to use an LLM as a general classifier for years. It's an incredibly common pattern for working with LLM judges or any sort of branched decision making workflow.

A lot of people who are a bit less familiar with the field saw the hype around Jev and presumed that the reason it was so exciting was that it was a fundamentally new interface for working with an LLM. And that additional excitement drove even more attention to Jev. But fundamentally, TypeSafe's announcement was that they found a particular architecture/training paradigm that resulted in a model for this particular interface that had incredible accuracy, very low latency, and for which they could offer inference at a super low cost.

I've not kept up with the flood of Jev clones that have been released, but I think this is just typical for any new component in deep learning that gets popular. There are an absurd number of open source autoregressive LLMs and fine tunes you can use. The thing that makes one more popular than the other is typically the general performance of the individual model.

But training a model for this purpose, or emulating the procedures described in Jev's papers, isn't something that would be beyond the capabilities of any lab. It's not an entirely alien architecture or approach.

The bigger question for TypeSafe as a company would be if other teams are producing Jev-like models that win on performance or cost. Like I said, I haven't followed the reports super closely, so no idea if that's the case or not.

That's not quite right. A recurring trend in ML is figuring out how to get an elastic interface for the the particular quirks of ML workloads (source: I worked on an open source one years ago for inference). Recently, there's been a lot of interest in doing this for agent workloads. Google recently released something called ax that is similar in spirit. The core of it is this: https://github.com/agent-substrate/substrate

At a high level, agent sandboxes have peculiar needs. Agents are really bursty, but also long lived. You need low latency suspend/resume calls. Checkpointing and recovery have some particular considerations. And naive approaches are often really wasteful, but over optimizing without harming durability, isolation, or consistency in performance can get tricky.

This is the new hot infra topic for agent swarms, for the time being. Whether that's impressive or not is up to the reader I guess, but it's a pretty involved project nonetheless.

I think the answers are "a lot" and "a lot more"? But what I'm saying is that Jev's virality isn't because people didn't have access to classifiers before it. Jev's general claims about capability and performance vs. cost would have been a very big deal 5 years ago too. In a vacuum, the idea that you can get a general classifier that is very accurate across any domain and on any modality with minimal latency and a very low price point is wild.

In early 2016, Clarifai's core product was basically just an image classifer exposed via an API. And at that point, they'd raised $40 million--the same amount as TypeSafe.ai/Jev--and they were experiencing viral growth among developers + signing contracts with a bunch of flashy logos. The demand was so high that Amazon launched Rekognition and Google launched their similar APIs to compete.

The AI hype cycle and the number of people thinking about using AI certainly puts more wind at Jev's back, but even in an alternative universe where we don't have contemporary LLMs, Jev's core claims would be remarkable and there would be a big market for it.

I don't know if that first part is true? Classifiers were/are one of the dominant applications of classic ML and neural networks, especially in production. Even today, image classification, object recognition, language detection, segmentation models etc are still super common.

I think the hype with Jev is just that, while structured generation is great, LLM judges tend to kind of suck for precise classification. And the more powerful the base model, the more accurate they can get, but they get increasingly expensive/impossible to finetune. It was specifically the latency/price point Jev offered vs. the general accuracy it claimed that generated all the excitement. Plus the promise of cheap calibration (tuning).

"Jev exists because LLMs exist" is kind of a truism, as Jev apparently is literally a Transformer model.

Yeah I'm actually less hesitant to try out Google open source projects than I am new Google products. I have no idea if this is accurate or just my impression, but I feel like I've been burned by the "killed by Google" meme almost exclusively on their software products, whereas there are plenty of open source efforts from Google that I think of as stable.

In addition to the ones you listed, I'd add the V8 runtime, Jax, Protobuf. Even some of their projects that wound up declining in market share (Angular, Tensorflow--both losing share to projects that wound up at Meta, ironically) are still actively maintained and pushed.

But I'm sure there's also a huge graveyard of open source projects they abandoned that just never hit my radar. Still, at least with their open source stuff, you can fork in the worst case.

I think this is basically the motivation behind MCP servers, so you're in good company.

But I've spoken with many people at companies who've decided to add an in-UI agent to their apps, and I don't think this trend will persist. Absolutely AI/agents will increasingly become a major component of interfaces, but the current common incarnation of "Clippy for X" feels like a kludged bridge between an app that was not designed from first principles for agents (and often was poorly designed for humans) and the impulse to be "AI native".

From my experience, some of the most successful niches for this sort of UI so far have been in apps that were already well suited to it. I'm thinking specifically about analytics/dashboarding/"this is a portal for you to query things easily" software. They already started with a lot of the elements you want: visible provenance of the agents actions via the queries it writes, a malleable interface that you already expect to be customizable and ephemeral, and most importantly, "navigation" that is genuinely difficult for many users (in the sense that "navigating" can mean "querying specific data"). The agent provides a ton of value to users and its actions are intuitive and legible.

But the agents you see right now in a lot of apps that do things like navigate you to the right page by... sending you a link, which you could have clicked from the navbar? Or worse, which hijack your navigation and throw you on a page you're unfamiliar with, and where you have no sense of place or how to make your way to/from? I don't think they're particularly long for this world.

Jev seems pretty cool! I just got access and have only gotten to do minimal experiments, but I love this general area of research and it fills a very real need.

I agree with you. I think the OPs pushback is emblematic of a larger reaction I've seen that is, at the very least, misinformed.

There are a lot of approaches that use a self-attention backbone for classifier-style outputs. You have structured generation libraries like SGLang and Outlines, but those basically give you guided generation on an autoregressive model. You also have a bunch of models that are non-autoregressive that try something similar. Older NLP stuff applies here, and there's newer stuff using diffusion transformers for this purpose.

But I don't think the Jev author has ever said that he's the sole human, alone in a vast sea of misguided researchers, who is interested in schema-guided classification? I think he said he found a novel way to train a model for this task that has much higher general intelligence at much lower cost than other approaches. Which is an exciting result with lots of applications if it bears out.

I think some people are just reflexively skeptical of anything that gets a lot of hype. Maybe that's fair. Things that are wildly successful and high impact also tend to get a lot of hype though, so it seems like a poor filter.

This is also a really common thing in ML specifically. We joke about getting Schmidthuber'd, which is when Jurgen Schmidthuber (sometimes correctly) announces that he or one of his colleagues actually proposed your thing 37 years ago in a Japanese linguists journal.

Statistical modeling, from simple classical stuff up to modern deep learning, just has this dynamic where the theory is rich and bottomless, but the actual components of implementation are pretty neat and compact. So for any given idea, there are probably 20,000 other people who have had the same intuition, just with subtly different application or implementation. Add in that depending on what your particular flavor of research is, you might name an almost identical implementation something completely different. And it leads to a huge amount of sour grapes whenever anyone's idea really garners attention.

If you listen to any podcast with a founder in the ML space who has been in it for long enough, they will invariably say at some point "We actually developed xyz over a year before OpenAI"

I understand the distinction you're drawing, but in my experience, it gets a lot blurrier as a project evolves. Caveat that I use LLMs all day and have for quite some time, so I'm coming from a positive perspective.

The danger for me in LLM code is the same as in writing, it's just that I'm not typically writing at the same scale as when I'm building something. The final piece when I'm writing is usually a message or a 1,000 word article at most. So I'm naturally going to analyze it quite intensely, because I can afford to. And I don't really use LLMs for this at all. I use them for things around writing (research, interrogating ideas, situationally specific stuff, mapping, visuals, publishing, etc.) And the code equivalent to an essay or message would probably be something like a single script, or the sort of thing I'd write as example code when I'm teaching. In those settings, again, LLMs can be helpful, but I'm still going to be really opinionated at a highly detailed resolution.

But a codebase is more comparable to a novel than an essay. Or more directly, the writing in a codebase is usually the documentation, which grows commensurately with the codebase. And the real LLM risk here is the drift that can happen over the course of many epics or "chapters" as the LLM writes "code that works but is imprecise and probably shouldn't work this way" or introduces weird new terminology that neither of us can precisely define. Worse, this usually becomes obvious down the line, and I have to parse through the verbose constructed world the agent has created to trace the issue back. That's a big cognitive tax, because I'm holding these weird parallel worlds of "How did the LLM's alien brain get here within the bounds of the contracts" and "What do I really want this to look like".

So I think it's fundamentally the same phenomenon, and we're all developing our skills around working with it in real time.