mrothroc

Hi, I'm the original author and I can clarify a few things.

The 543 hours are the agent compute hours, not me at the keyboard. The pipeline runs autonomously, the agents execute in parallel, and the gates verify the output. Most of the prompts are agent-to-agent, not human-to-agent.

On the timeline: I have a BSCS (1995) and MSCS (1997) with a specialty in distributed systems. I actually worked my way through school doing this work so I didn't need loans. Let's call it almost 35 years.

The terminology has evolved but the architecture hasn't changed as much as people think.

Thank you for your feedback. These are fair points.

I get that "top performer" is off-putting. You're right that authority has to be earned in the text (and I hope I do that), not declared.

On the structure: yes, it's a novel format and I can see how that would be hard to parse. It won't work for everyone.

Both of these are artifacts of trying to blend research into the modern social-media driven world.

I'm the author of that post. Thank you for your feedback.

The production code is proprietary work for clients, so I can't link to it directly. But the tooling I built to support the pipeline is open source: the log analyzer that computed these statistics.

There are a couple of other in-flight projects I will open source soon, created by this process, but they aren't out yet.

The research page is about the methodology because that's what generalizes. The specific microservices I ship are just microservices.

I did the same with my own orchestrator. That's where I get my data.

It's amazing the power a simple workflow with automatic gate enforcement brings to agenting coding.

I created my own framework. Long ago it started as shell scripts that I used in conjunction with aider. It was a very manual process.

It's grown over time to be a full MCP and CLI with stages and gates defined in YAML. I was thinking about open sourcing it but since the code grew organically I would need to do extensive cleanup to make it presentable.

But I do walk through the process on page 9: https://michael.roth.rocks/research/trust-topology/#9

Agreed that full consensus is overkill.

But I think the coordination problem is subtler than version control implies. In the (plan, design, code) pipeline they aren't collaborating on the same artifact. They're producing different artifacts that are all expressions of the same intent in different spaces: a plan in natural language, a design in a structured spec, code in a formal language.

Different artifacts which are different projections in different Chomsky levels but all from the same thing: user intent.

The coordination challenge is keeping these consistent with each other as each stage transforms the prior projection into the new one. That's where the gates earn their place: they verify that each transformation preserves the intent from the previous stage.

I've been running a multi-agent software development pipeline for a while now and I've reached the same conclusion: it's a distributed systems problem.

My approach has been more pragmatic than theoretical: I break work into sequential stages (plan, design, code) with verification gates. Each gate has deterministic checks (compile, lint, etc) and an agentic reviewer for qualitative assessment.

Collectively, this looks like a distributed system. The artifacts reflect the shared state.

The author's point about external validation converting misinterpretations into detectable failures is exactly what I've found empirically. You can't make the agent reliable on its own, but you can make the protocol reliable by checking at every boundary.

The deterministic gates provide a hard floor of guarantees. The agentic gates provide soft probabilistic assertions.

I wrote up the data and the framework I use: https://michael.roth.rocks/research/trust-topology/

Non-coders often think all engineers do is write code. They don't realize how much more hours are spent on making sure the code we write is correct, from many angles. Functional bugs? Easy to maintain? Cost optimized? Meets user expectations?

When they have a machine that cranks out code, and honestly pretty good code, they think it's the same thing.

Eventually though I suspect many people will discover what many studies over many decades have shown: the most expensive part of software is maintenance.

This will eventually be the problem for your client. If I were in your shoes, I'd probably start laying the foundation for that. If they're willing to bear the cost, then fine. If not, then ultimately you're set up for a clean exit when you can no longer get the software to run as they want in a timely fashion.

I see this has been updated by the user showing it is their own tool doing the damage.

These things happen. They happened before coding agents, they happen now. I've done plenty of damage with my own ten fingers on the keyboard without any help from an LLM.

This is exactly why I develop on a Mac with Time Machine. It has saved my bacon many times. Both from things I did and from things Claude did. I've had several recent incidents that went like this:

"me: Claude, did you delete X?" "claude: Yes, sorry, I shouldn't have done that. I can reconstruct it." [Narrator: no, claude cannot reconstruct it.] "me: Should I just restore it from Time Machine?" "claude: Yes! That's perfect!"

I swear I can feel a sense of relief from Claude when I tell it I can just restore from backup.

It's the whole tool that's important, not so much how you get screenshots. That's what I'm saying: this is headed in the right direction, it just falls a little short of what I do, where I get tons of value over and above just playwright (or whatever gets the screenshot).

The critical part is that viewed at a high level, this method tests something different, which means it catches different errors.

Everyone is comparing this to Playwright but it's solving a different problem. Playwright checks structural properties, like does element X exist, is it visible, etc. That's useful but it can't tell you whether the page actually looks right.

I built something similar that takes a screenshot and uses a multi-modal LLM to evaluate it against a design mock. It catches a completely different class of error. The DOM can be structurally perfect and still look nothing like what was intended. Colors wrong, layout shifted, spacing off, components overlapping. No amount of DOM assertions will catch that.

These are two different kinds of gates: structural which are fast and deterministic, and stochastic which are slow but catch things that are completely different. There is very little overlap between the issues, and you want to catch both.

That way I can invest a lot of time getting the mock just right, then let the agents "make it so".

I do this across several different codebases, all event-driven microservices. Some greenfield some brownfield.

My strategy is to have a central spec, typically protobuf or openapi, and every service has a make target to generate code from that. The dependencies stop being in someone's head and start being in the spec. I had this pattern long before coding agents because it helps human devs, too.

The benefit is that my CI process can deterministically check if a) someone changed the spec or b) the service code doesn't compile or lint when built against the freshly generated code. This is a hard, enforced gate. If it fails, either it needs manual review or it gets sent back to the coding agent to fix automatically, so humans don't waste time looking at trivial issues.

The agents can move as fast as they want within a single service. The spec gate catches cross-service breakage before it deploys.

Everyone keeps saying 80/20 but that undersells what's going on. The last 20% isn't just hard. It's hard because of what happened during the first 80%.

When an agent takes a shortcut early on, the next step doesn't know it was a shortcut. It just builds on whatever it was handed. And then the step after that does the same thing. So by hour 80 you're sitting there trying to fix what looks like a UI bug and you realize the actual problem is three layers back. You're not doing the "hard 20%." You're paying interest on shortcuts you didn't even know were taken. (As I type this I'm having flashbacks to helping my kid build lego sets.)

The author figured this out by accident. He stopped prompting and opened Figma to design what he actually wanted. That's the move. He broke the chain before the next stage could build on it. The 100 hours is what it costs when you don't do that.

I use prompt templates, so in the first version of my analysis script on my own logs I looked for those. However, to make it generic, I switched to using gemini as a classifier. That's what's in the repo.

Nice, I've been working on the same problem from a different direction. Instead of analyzing sessions after the fact, I built a pipeline that structures them. Stages (plan, design, code, review, same as you'd have with humans) with gates in between.

The gates categorize issues into auto-fix or human-review. Auto-fix gets sent back to the coding agent, it re-reviews, and only the hard stuff makes it to me. That structure took me from about 73% first-pass acceptance to over 90%.

What I've been focused on lately is figuring out which gates actually earn their keep and which ones overlap with each other. The session-level analytics you're building would be useful on top of this, I don't have great visibility into token usage or timing per stage right now.

I wrote up the analysis: https://michael.roth.rocks/research/543-hours/

I also open sourced my log analysis tools: https://github.com/mrothroc/claude-code-log-analyzer

Senior review can definitely help, regardless if the code comes from a junior or an LLM. We've done this since the dawn of time. However, it doesn't scale and since LLM volume far exceeds what juniors can do, you end up overwhelming the seniors, who are normally overbooked anyway.

The other problem is that the type of errors LLMs make are different than juniors. There are huge sections of genuinely good code. So the senior gets "review fatigue" because so much looks good they just start rubber stamping.

I use an automated pipeline to generate code (including terraform, risking infrastructure nukes), and I am the senior reviewer. But I have gates that do a whole range of checks, both deterministic and stochastic, before it ever gets to me. Easy things are pushed back to the LLM for it to autofix. I only see things where my eyes can actually make a difference.

Amazon's instinct is right (add a gate), but the implementation is wrong (make it human). Automated checks first, humans for what's left.

The disposition problem you describe maps to something I keep running into. I've been running fully autonomous software development agents in my own harness and there's real tension between "check everything" and "agent churns forever".

It'a a liveness constraint: more checks means less of the agent output can pass. Even if the probabilistic mass of the output centers around "correct", you can still over-check and the pipeline shuts down.

The thing I noticed: the errors have a pattern and you can categorize them. If you break up the artifact delivery into stages, you can add gates in between to catch specific classes of errors. You keep throughput while improving quality. In the end, instead of LLMs with "personas", I structured my pipeline around "artifact you create".

I wrote up the data and reasoning framework here: https://michael.roth.rocks/research/trust-topology/

Everyone is circling around this. We are shifting to "code factories" that take user intent in at one end and crank out code at the other end. The big question: can you trust it?

We're building our tooling around it (thanks, Claude!) and seeing what works. Personally, I have my own harness and I've been focused on 1) discovering issues (in the broadest sense) and 2) categorizing the issues into "hard" and "easy" to solve inside the pipeline itself.

I found patterns in the errors the coding agents made in my harness, which I then exploited. I have an automated workflow that produces code in stages. I added structured checks to catch the "easy" problems at stage boundaries. It fixes those automatically. It escalates the "hard" problems to me.

In the end, this structure took me from ~73% first-pass to over 90%.

Yeah, this is what happens when there's nothing between "the agent decided to do this" and "it happened." The agent followed the state file logically. It wasn't wrong. It just wasn't checked.

His post-mortem is solid but I think he's overcorrecting. If he does this as part of a CICD pipeline and he manually reviews every time, he will pretty quickly get "verification fatigue". The vast majority of cases are fine, so he'll build the habit of automatically approving it. Sure, he'll deeply review the first ones, but over time it becomes less because he'll almost always find nothing. Then he'll pay less attention. This is how humans work.

He could automate the "easy" ones, though. TF plans are parseable, so maybe his time would be better spent only reviewing destructive changes. I've been running autonomous agents on production code for a while and this is the pattern that keeps working: start by reviewing everything, notice you're rubber-stamping most of it, then encode the safe cases so you only see the ones that matter.

I've been running a multi-agent setup for quite a while to do software development. I set up a workflow with agents at each stage, spec->plan->design->code->review. The key thing I learned was that the arrangement of the checks between agents matters more than which model you pick for any one step. Most failures were omissions that a gate between stages catches.

I ended up building my own for this. SQLite backend, breaks work into stages with gates between them. A gate checks each handoff before the next stage starts. Does the code match the spec, did the tests pass, that kind of thing.

I've been running it with Claude Code for about four months now. What I didn't expect was how much the gates matter relative to the model itself. Most of the failures I see aren't hallucinations, they're omissions, and a structured check catches those easily.

I wrote up how I did it and how to start building one yourself from markdown and a shell script: https://michael.roth.rocks/research/543-hours/#10

The checkpoint pattern you describe is exactly right. I've been dealing with this as well. Instead of vibe coding, it's vibe system engineering and I don't care for it. So I thought about it and came up with a framework to describe and reason about different pipelines. I based it on the types of LLM failures I was seeing in my own pipeline (omissions, incorrect, or inconsistent with existing stuff).

I wanted something I could use to objectively decide if one test (or gate, as I call them) is better than another, and how do they work as a holistic system.

My personal tool encodes a workflow that has stages and gates. The gates enforce handoff. Once I did this I went from ~73% first-pass approval to over 90% just by adding structured checks at stage boundaries.

My hope is that we can have a common vocabulary to talk about this, so I wrote up the data and the framework that fell out of it: https://michael.roth.rocks/research/trust-topology/

I'm old enough to remember that engineers researching distributed systems had the same challenge. Everyone was trying to build 100% reliable nodes, which is impossible. Then Lamport came along and showed you could actually achieve your goal at the protocol/system level. What you're describing here is a workflow or pipeline, which is the analogy. As the LLMs produce artifacts, you have gates that verify the output deterministically. If the LLM breaks a rule, you either throw it out and reroll or you give it the feedback and let it revise. I do this in my own tooling and I get great results. One thing from the data: they are often pretty crap at revising, spending ridiculous time/tokens in a revision loop. I'm trying to find the right balance of reroll/revise myself.

My experience is similar to yours: LLMs can write excellent code, though you really have to drive them the right way. I use a harness to drive long-run autonomous agents to create production code. (Not open source, but it is an actual product used by companies.) The key is understanding how they fail, then driving them in a way that sidesteps this. If you let them run too long, they become self-contradictory. However, if you break long work into discrete chunks, then they can still fail, but it changes: they forget things. But that is a much easier thing to catch, because you can use things like lint or even a simple regex for "// TODO" to find them. Once you set up your pipeline to orchestrate the agents so the errors become easily detectable, and you have gates that check for those errors, the quality goes way, way up.