Agents fail like engineers, so I review them like engineers | Hacker News Reader

Agents fail like engineers, so I review them like engineers

blog.greenflagged.dev

15 pointsAzkar5 comments

With these multi-agent workflows, I often wonder when coordinating the agents and reviewing their findings starts taking more time and effort than working with a single agent. Do you have a good feel for when (if ever) that happens?

Relatedly, I'm seeing more and more people who are opening and closing a hundred pull requests a day, and outside of areas like performance profiling, I have a hard time understanding how that is being achieved. Are you finding yourself naturally gravitating toward that type of velocity and having to push back on it? Or do you see these types of workflows eventually allowing you to achieve a similar pace of change while still maintaining an emphasis on human reviews?

> I often wonder when coordinating the agents and reviewing their findings starts taking more time and effort than working with a single agent

I think this is a great question, and it probably needs to be re-evaluated every time models change. I added each of these to my workflow because I found that as my project grew in size and lines of code, single agents were missing large pieces of context and needed to be fed a lot of guidance. You can also determine how many/which type of agents to spawn based on the size/risk of the change. Ultimately it's probably a trade-off between speed/safety. For short term projects, a single agent is fine.

> I have a hard time understanding how that is being achieved. Are you finding yourself naturally gravitating toward that type of velocity and having to push back on it?

I can see it. I do have to push back on the urge to want to do more things in parallel. However, the more I use these agent workflows, the more I trust them, the more autonomy I give them. I have no problem passing a well written ticket to an agent and trusting it to build/test/merge/validate the change in prod.

saw the skeptic is pinned to opus. wondering how it'd do with non-claude models as reviewers. and what does a full review run cost you, max plan or per token?

I use the max plan, I don’t usually check the usage for a full build so I don’t know exact token usage.

Looking at my recent usage, it’s the second most after the feature orchestration itself. It will also depend on how many claims it has to verify and what I’m building. It’s the most valuable piece of glue that I’ve found when passing things between other focused agents.

My experience is with Claude, but I expect other frontier models would work just as well.

I like how you break down each type by how the agent compares to human behavior.

Granted I don't know the specifics behind training LLMs but at the end of the day they are typically trained on data that originated from humans.

So in some ways it's better to try and understand their contributions to our work from that mindset than just saying they are computers/text generators.

Reply on news.ycombinator.com