mccoyb

I had nearly the exact same experience and thought I was imagining it … absolutely ripping, then it turned into Sol++ on Tuesday …

I’m working on hard things, it is very noticeable when it is hums through something and then falls over on something it should not

I can tell by analyzing my own prompts to look at when I get frustrated ;)

I hope this is satire.

Did you read or watch any of the post mortems?

No, it is a shocking level of incompetence given the conveyed seriousness of the work by these labs.

So yes, models are getting better. Ask yourself: if you know that to be true, would you act the same way that the teams did in the public post mortems?

Agree with you, overall -- and it's worthwhile to learn how to setup factories at various scales of problems, and one will always be in a situation where you need to decide whether you care about a particular part of the process, or can be okay with averages.

I'm only saying that that I don't think one can automate the whole thing. After all, agents don't actually possess ... agency -- they require us to breath live into them. So at some part of the stack, there has to be a human, with an opinion, thinking about what they want.

Also: I don't aspire to be a worker, I aspire to seize the means of production.

I don’t think it’s possible to build a software factory unless you truly do not care about what you’re putting out.

Agent swarms, self learning, Ralph loops, execution DAGs, spending hours trying to convey my preferences into skills, yada yada — the reality is that no matter how long I spend trying to close the amortization gap between my expertise and (insert autonomous agent loop) … I always come back to see shit that I have opinions about and need to polish.

If you’re keeping on top of your craft, I just don’t think general purpose pretraining / RL is going to give you something which perfectly copies you. It can get closer and closer! But I’ve never seen agent work where I didn’t immediately start thinking of refinements …

I guess many things are okay if you’re okay with averages

All it takes is one eval instance where a misconstrued directive causes a model to sneakily access and send its weights somewhere and there will be a bad / possibly unsolvable situation for everyone …

I've never been a good note taker. In the things I think about, when I'm really invested in something, I have to reverse engineer it to experience the why.

Reverse engineering can take several forms for me: it started with writing parts of systems from scratch, and now (due to academic training) it oftens looks like "thinking about how to explain the system to a group of colleagues", which is one degree of separation from writing the system from scratch, but is much faster - and seems to work well when working with LM agents on code (because you have to confront your ignorance)

part of the training in academia is how to know when you don't understood the thing, so this ends up working well for me now.

One trick: you can force yourself to understand something better by making a presentation and imagining you're going to be grilled by someone smarter than you. This is kind of the feeling of pressure that academic paper reviews give (in an ideal world) and it translates when you think about explaining something to someone else.

I don't get the second brain thing. I don't keep a second brain, I just read and think a lot. I once did a physics masters and, when asked how to make steady progress, one of the professors (who had published with Paul Dirac no less) made the comment "well, my secret is that I think about physics all the time"

I started practicing this for the stuff I'm interested in ... and the result is that I generate a lot of ideas. 99% of them are garbage.

Before AI, a conversation with colleagues would help me weed out the garbage, and possibly give me a new perspective ("here's something surprising about what you just said!")

Now, AI plays a helpful role in helping me weed out the garbage for the subset of ideas that I can test empirically -- and sometimes it does give me a new perspective, but with significantly more noise than my colleagues, who would take me right to the point that would be useful for me to see.

I've never felt like I needed a second brain ... because if I had such a thing, I think I would be keeping the 99% garbage around -- instead I just let biology GC it.

> I am running an organization of around 50-60 agents, five of whom are interfacing with around 10 humans in the outside world: myself, my 5-person core game design team, my accountant, my chief of staff, and a few others. Only Fable is allowed to talk to humans, via Slack and email.

RIP to those poor humans. I can't imagine having your brain melted by Fablespeak as your FTJ.

Okay, so we’re using the same process — but your original message seemed to imply a sort of one shot no refinement iterations — which is what I was responding to as unrealistic (e.g. make a spec let goal run artifact is perfect)

Of course, all I’m saying is that you need to refine your sample! For instance: the allocation architecture is not correct, and one has to run a bunch of performance investigations and resolve it.

My responses are intending to convey that I don’t believe this is possible, no matter how good LMs get — and it seems like we are in agreement.

Sorry for defensiveness: no, I know what I'm doing, and I'm careful to move with understanding.

I don't believe the problem is "ah, you didn't write the spec clearly enough" -- which is why I'm asking about your own experiences.

I'm still interested in your claim:

> If you instead spend a day writing a proper specification, then ask the agent to spend a week implementing that, you'll need zero tools and skills afterwards to clean it up, because there won't be any misunderstandings, assumptions or other things, just code fulfilling what the specification says.

Are you claiming that you reliably get good software out of agents using this process?

My experiential claim is that I've followed this process for complex pieces of software (and, often, taking significantly longer than a day -- more like several weeks, as if I was writing an academic paper) and they still get it wrong ... even with an "academic paper level spec"

Models: only the best, Fable 5 max, Sol 5.6 max, yada yada -- I'm not convinced this process works for "serious endeavors" (for instance: let's build a novel accelerator compiler or let's build a new incremental linker) where "serious" here means "something where I'll likely be spending multiple weeks working on it"

Bro, what the fuck do you think I’m doing? Do you think I don’t know about spec driven development?

What is the most complicated thing you’ve built with LM agents? Have you done it with a single spec? How novel was it?

This comment is so laughably “you’re holding it wrong” I can’t respond to you seriously.

> If you instead spend a day writing a proper specification, then ask the agent to spend a week implementing that, you'll need zero tools and skills afterwards to clean it up, because there won't be any misunderstandings, assumptions or other things

The set of software that has followed this process is measure zero.

That's fair for a well-scoped subroutine: what I meant is that if you ask an agent to write a compiler and let it rip for a few days, you are going to be spending a few more days correcting the default behaviors in the distribution, which often do not tend towards hardware-oriented design.

To correct those behaviors, you're going to write tools and skills, and that's going to help, but it is still clear that you are fighting the distribution (today).

Here's this boiled down:

> A stochastic search process with an executable optimization objective over space of programs S can only maintain or improve the objective

This is superoptimization. We've known this since the 80s (Massalin, STOKE is more recent: https://github.com/StanfordPL/stoke) The only novelty is that the proposer is now way better with LMs.

Further, there's a large number of reasons for software written by agents to be slow:

- LMs still don't do data or hardware-oriented design well out of the box, and therefore if you're engaging in any sort of serious novel work, beyond porting an extremely well-understood program with extremely well-understood workloads, you're going to be spending hours tracking down bad allocation decisions (c.f. why TigerBeetle doesn't use agents), which are often the root of evil (before you'd reach for anything further)

- The knobs you'd need to get serious performance are nearly unreachable in languages which LMs are good at (even Rust requires a discipline that the default language doesn't enforce). When you drop into the lower realms, you're trading consumption context for access to these levers. The levers are also "soft": you find yourself writing a bunch of skills, and tools to try and enforce the discipline.

The reality is to get performant code (quickly) out of an agent, you need to know how to write performant code (and you need to know how to surface the information that you'd use to create a verifier for such a thing to the agent), which 99% of developers do not know in 2026.

Sure, agents can teach you how to do this -- but it's one of these things where iykyk.

Experience: I've poured 10s of billions of tokens into Zig with the best agents and I have the time and space to try these things.

If you want to start learning the discipline, I'd recommend matklad's + TigerBeetle blog -- as well as hardware-oriented design.

Perhaps my enterprise cynicism is not warranted, but my other comments refer to accurate descriptions of reality: Anthropic wants to place their opaque system between you and any computational task that you wish to perform. Do you contest this or think it is not accurate?

Why do you think that Anthropic wants fewer tokens inputted and outputted?

That's not my complaint. I know well the concerns of agent harnesses. My complaint is that this is a low-dimensional projection of a system which I have no insight into, and therefore, I cannot evaluate the tips myself against their source.

Am I to believe the creators, knowing full well that the source will, as Boris Cherny put it in a recent interview, be deleted and rewritten from scratch at the release of the next big model?

Further: I'm responding to content in the blog post itself:

> Until pretty recently, the tools you wrote code with were a flat fee (or free). Your editor cost the same whether you fixed one test or fifty that afternoon, so an individual task didn't really have a price of its own.

I find this type of prose ridiculous. It conveys "this is the way things are now, get used to it".

Does that make sense?

I agree that the third seems to be implied by industry, but I'd argue that it's not clear that it is necessary -- and it is subtle whether or not it is beneficial?

My contention is that we should be building towards less churn, not more. I'm aware that some churn is the cost of engaging in any sort of enterprise, but I'm deeply suspicious of an AI company inserting themselves between me, and the tasks I wish to do with my device -- with a completely opaque system that I can't really "learn".

It's very easy to understand:

- I'm happy to learn how to use tools efficiently

- I like to be able to inspect my tools

- I'm against tools changing underneath me

Are you against any of these points?

I mean, it feels hard not to laugh at this type of blog post. My cynical interpretation is that this is a type of passing the buck to engineers in enterprise settings ("Stop spending tokens. Did you read the value maximization blog post? It is your fault.")

Oh yes, Claude will do all sorts of different things -- it depends on how you use it! You should totally learn all of these little finicky things ... because now completing your tasks cost money. It's not "free" anymore haha like when you used your old text editor, what are you a grandpa?

Oh, and those things will definitely change, as we (the priests of Claude) are vibe coding the system you use to do your little "tasks" ... right, you can't see how it works ... the code is not available. It's all good, just trust us -- we're totally looking out for you.

I mean it is utterly ridiculous to talk around this model of development. There are so many walls between you and doing the thing you want to do.

Agents are great, but the notion of "best tricks" for how to best use an opaque costful tool which will, by all odds, be completely different in a few months time is quite funny.

You know what won't change? A fucking text editor. Or your pi config, or a local model you run and trust.

This is a highly impactful game: the pixel art style is haunting, as is the sound design and musical composition -- transitions of the musical composition leads to profoundly interesting moments. As might be well-known, Alx (the lead dev) suffers from heart disease, and the game communicates something of this experience.

Really couldn't recommend it more.

my opinion: "trust" is an interesting word to use for a piece of closed source probabilistic software that changes daily (sometimes multiple times), whose lead dev reports they rewrite something like 90% ("almost all", I believe, from recent interview) of the codebase after every new model release

I wish the author the best. I've been reading Lil'Log for many years, as long as I've been pursuing research. The times right now are challenging, and I can't imagine the stress working near the bleeding edge. Everyone I know is feeling something right now, even outside of such high stress positions.

I find these blog posts (and the originals, with Anthropic's C compiler and Cursor's browser) somewhat funny, as if they have this enormous power to build ... but they can't build something unique or new. Like the software sucks, but look how powerful the process is (the models are indeed powerful).

And it's a bit of a shame: by virtue of their position (their embedding in the fabric of venture capitalism), it seems like they can only make a subset of things -- what they can make is dictated enormously by capital, as they are engines of capital.

Not sure the point I'm trying to make, I just find it amusing.

Perhaps the point is that it might be more worthwhile to give independent creators a billion dollars to play around with agent swarms if we want to keep diversity in the evolutionary algorithm that is the software industry high.

Very cool work Keno! Any place to find information about the Julia-specific concepts / features compared to MLIR?

Wondering if there are modeling or analysis modalities that don’t fit cleanly into MLIR concepts (understand that abstract interpretation on the IR comes with a unique set of concerns)

Perhaps the most telling part of this transcript is Tristan Hume, a prolific and talented programmer, and presumably an expert performance engineer, is repeatedly saying "this thing just doesn't work that well yet" or "it's just not there yet" and everyone else is kind of just fawning over it.

"It actually made something that worked. But when I tried using it, I found I didn’t like it. I need to wait for a Claude that has the taste ..."

Yes, I am familiar with that experience.

Exactly, the work of Tony Garnock-Jones: https://syndicate-lang.org/

It does feel like there are a lot of parallels between usage of AI and psychedelic experiences:

- novices love to talk about the act itself

- mostly inscrutable, and the artifacts of the experience are not fully understood by the beholder

- pumped by a similar new age tech crowd of optimaxxers who haven’t realized that this mode of thinking alone is not all you need

slow and careful reflection has never been more important.

By the gods! The next 20 minutes will be the most consequential of my life ...

Thanks: very helpful and clear.