Hacker News Reader

Comment by andai | Hacker News Reader

Can you elaborate on this? Especially the tooling.

I tried something similar and I remember it was still pretty dodgy in February.

my stack in a sentence: refine the docs/prompts/skills often, that's your biggest job, use both frontier labs models reviewing each other, don't solve individual problems only the systemic ones (set standards strategically, don't define tactics)

If I had that many tokens/dollars I would be running canaries and adversarial verification in prod based on e.g. traffic replay, live fuzzing, all kinds of things to build confidence without direct human line-by-line review. If I had $100k to spend next month I could probably get through it, I'm running $2500+-api-equivalent a week at this point and I feel very token limited. Will be time for a 2nd or 3rd subscription soon for both labs I think.

Fable was a revolution, still learning how best to use it, 5.1 felt like a notable upgrade. At this point I launch a workflow with 10-20 minutes of interactive setup (and even that I feel might be too much), it runs for hours, and the PR is trivially mergeable (I still review every line, but 95% are just merge, maybe 4% are feedback needed, 1% are thrown away and regenerated, which implies I'm being insufficiently ambitious)

Thanks!

> If I had that many tokens/dollars I would be running canaries and adversarial verification in prod based on e.g. traffic replay, live fuzzing, all kinds of things to build confidence without direct human line-by-line review.

This part jumped out at me. There's something to watch out for here.

I recently had a funny experience. I delegated a major feature to an agent.

It turned out that it had implemented it precisely backwards, in a way which was pointless and which made things worse.

But it had written countless tests for the feature and all the tests were green.

I realised in that moment that even formal verification would not have helped, because it would simply have written a mathematical proof of the correctness of the incorrect feature...

Yes, you need some kind of other source of truth. I think the best way to get that is to do clean room development with a different agent, but ultimately if you give them the wrong idea they'll do the wrong thing.

The other thing I do, not as much as I should, but it's very powerful, is to generate spikes and deliberately throw them away to understand how to prompt better. Like I generated a swift version of the react native app I'm working on, and Alloy provers for the state transitions. None of it is production quality but getting great results that way is useful to scope future work.

If you've managed people, these are all familiar problems. I found you need much more than a functional specification, you also need motivation, background, related work, ideas tried, etc., because those help disambiguate the right path in the inevitable situation where your original task description is unclear or conflicts with itself.

Yes I'm using the entire consultancy stack - define values, etc, and work your way down the "where do these not match reality on the ground and need change", but for little robot people instead of (arguably less messy) humans.

> I think the best way to get that is to do clean room development with a different agent

What is this about? Could you give an example?

So, if I'm building, say an iOS app, I have a) the code itself (swift/react native), b) the internal test suite (unit tests), and c) the e2e test suite with something like XCUITest that drives the physical development phone, plus d) the backend (which has similar tests but is basically a mirror, so I won't elaborate)

So when you're creating a (and hopefully b), you use one model with one context, and then you might use some other model to implement c. I kind of round robin the models and present them with seperate context - so for example I don't build a, b, and c together, I build a plus some b, then later go for a pass over b and c together, then maybe I used c to drive improvements in a, and I vary between openai and anthropic models as I do.

Heres an example set of minimal prompts:

a-focus:

implement feature <x> based on #ticket in github, be sure to reference the engineering standards documents and the swift and react native skills as needed

b-focus:

improve test coverage in the repo for <subsystem z> to ensure that <feature x> is covered completely, and fix any outstanding gaps or omissions in that feature as you go (standard references above)

c-focus (possibly in a separate repo):

You are creating a black box xcuitest to drive a physical phone for testing <feature x>, here is the user specification and known issues, create failing tests for each known issue and an overall robust suite to ensure any user facing or ui issues are caught

Pretty much goal + task + dependency infrastructure to help avoid drift during long runs, especially across compaction boundaries. I have spent a lot of time doing automation with models at various strengths including some of the early open-weights models (Llama, Mistral, etc) so I have a pretty good feel for how to steer productively, there isn't any deep magic just scaffolding built out of reading a lot of traces and debugging stuck agents.

Reply on news.ycombinator.com