Loop Engineering: Optimizing Code Generation and Verification | Hacker News Reader

Loop Engineering: Optimizing Code Generation and Verification

bradleygibbs.dev

12 pointsbradleygibbs13 comments

https://www.pangram.com/history/664f8cc4-cfcb-40f4-8c81-e468...

Don't waste your time. (s)Looped!!

Sure, could probably be made shorter, but the content was more valuable than some pangram score. I found it useful to read as it discusses a problem currently facing teams.

Still worth a read.

Did you read it or did you have your agent read it?

It seems people are somewhat converging on the same process.

I setup https://github.com/schancel/software-factory as an interim process for how I work. It burns through tokens quite quickly -- but it gets work done with minimal intervention from me. The skills create tickets, dependencies, spawn agents in parallel where possible -- manage claims through 1 github account on tickets to track which agent is working on what; and provide contexts where agents need to restart or change models.

It hypothetically right-sizes tickets for sending various items to right-sized models if you use the latest frontier models as a coordinator.

But again, still iterating as I find issues.

Interesting repo!

No followers and 1 star, so, I'm assuming that you mean that process is what people are merging on, not necessarily this particular repo?

There are some similarities between this and what we've built (as described in the article I posted). One thing I noticed is that this workflow is based largely on agent skills, which are guidance/advisory. Sometimes agents follow them. Sometimes they don't.

We prefer deterministic runners for the process overall. We've also found that passing instructions to agents in their initial task context just prior to spawning them makes them much more likely to adhere to those instructions.

Deterministic everywhere possible. Firmly bounded everywhere else. The design conformance contract should be frozen before starting code generation and the evaluator must exist independent of the build loop.

Brad

Author here. This is a detailed (8,000+ word, plus 2 companion pieces) walk-through of building loops for code generation and verification, complete with artifacts and detailed explanations to allow you to implement any of these techniques.

We walk through building: - an evaluation contract, evaluator and stopping conditions to minimize bugs - establishing an environment and priming AI to produce production-quality tests and code - detecting and correcting common AI failure modes - building a learning loop to monitor and improve all phases.

I'd love to get your feedback!

Brad

> (8,000+ word, plus 2 companion pieces)

I appreciate the idea, but I doubt you wrote or actually read the whole thing. There's no point in trying to impress people with complexity. The art of writing is to convey your message with the least amount of text, while retaining maximum information, and making it easy to read.

I could've written what I wrote above with 8,000+ words too.

I think it's just too much. Honestly it feels like the whole thing could be a tight 100 line bash script. The diagram at the start didn't look too complicated

it's full of slop, even if there is something you were hoping to convey, it's lost in 8,000+ words. In this LLM world, less is more.

Thank you, Mr. AI. Seriously, this slop is worthless and you burned all those tokens for nothing.

You are not the author, but merely the driver of the slop dozer.

Did I use AI to help write this article? Absolutely. I use AI 10-14 hours day, seven days a week to do all sorts of things, including trying to get a code generation and verification app off the ground. It would be unnatural for me not to use AI to research and help think through this article.

Do I know how to use Pangram, too? Absolutely. I got my Pangram merit badge months ago.

So, why didn't I make sure I got a 100% human-written score on Pangram before putting this out there?

I spent weeks reading what others had to say on these topics--Lilian Weng, Addy Osmani, Han Lee, Simon Willison, and a number of arxiv papers. I felt I had some things to add to the conversation. A slight twist here and there, maybe some more detailed information about how to implement these ideas than I'd seen elsewhere. For example, I see people writing about the model eating the harness, but very few people countering with "but AI failure modes".

I feel like there are a lot of holes and gaps in what I've been able to piece together from others' writings, largely because each article is only covering a portion of the entire process. The reader is left to fill in the blanks and connect the processes.

I have firsthand experience filling those gaps and holes I wanted to share. I felt this needed to be a single article, against the advice of multiple AI writing partners, because that's where the devil lives, in the details. Without a single piece covering the process from end-to-end, the reader is left to fill in the gaps themselves by triangulating multiple excellent articles. I've done that work. I wanted to share what I've found with others to try to start meaningful conversations about best practices for code generation, code verification, and learning loops that can optimize the entire process.

So, call it slop because Pangram told you you could. But I spent ~20 hours just researching and writing the article, not including the 100's of hours I spent writing code generation and verification pipelines.

I remain eager to engage with anyone on the contents of the article.

Brad

Reply on news.ycombinator.com