Ask HN: Beyond traces, how do you know your agent did the job? | Hacker News Reader

Ask HN: Beyond traces, how do you know your agent did the job?

5 pointsrstagi3 comments

I keep going back and forth on this.

Traces and logs tell me what the agent did: every tool call, retry, token, and step.

That's useful when something breaks. What I'm less sure about is whether that tells me the agent actually did the right thing.

A run can look clean and still miss what I wanted. Or it can "finish" without the outcome really being done. So finished and succeeded aren't the same thing.

I haven't made up my mind how to check for that, but I'm experimenting a lot with extracting intents from my agent and correlating among them.

Curious how you do it:

- Do you capture intent and outcome explicitly (e.g. giving a tool to the agent to "declare" it), or infer them after the fact from the traces?

- What kind of "questions" do you "ask" your traces?

View on HN

Unit tests? Or plain ordinary testing the same way you would do for other software? (e.g. even if it not software the same idea can apply)

I have a QR code redirect system that generates unique codes for business cards, flyers and such. Previously it generated PNG files but I wanted to load a PDF template, merge in N unique QR codes, and make a new PDF with N pages. I talked it over with Claude, who offered me a test plan although I didn't ask for it -- it was roughly the same test plan I would have made myself. Like I made some flyers, checked that they looked right, checked that the QR code scanned and redirected to the right place, etc.

Claude did a pretty good job, i had to do some visual tweaks, add something manually to the back end to accomodate it and make some little fixes. If I had a junior programmer do the task it would have been the same story.

Whatever you are trying to make by any means whatsoever you are going to have to convince yourself you like the result in the end.

I cant code, but Im building stats tools for trading on my own with several AI agents on three machines. I dont let a agent approve its own work, so a second agent on another machine recompute the results using only the written method without looking at the first agents code. Today that caught 2 numbers that only worked under assumption nobody had written down, an import that crashed above about 150K rows and an outdated threshold in the text. All the first agents tests had passed. The traces tell me what it did, but I still need another agent to check the numbers.

we built zeroshot.sh to let you mechanically enforce this flow (and any other topology), want to check it out? :)

Reply on news.ycombinator.com