Hacker News Reader

Comment by vetronauta | Hacker News Reader

If I understood correctly, the author had the agent perform manual mutation testing (write code, write passing tests, manually change the code to see some tests fail, then revert the change), rather than automated mutation testing. Why not use a mutation-testing framework and consider the build failed if a certain percentage of mutations are not killed?

The article claims that the agents didn’t actually use TDD or mutation testing. While it’s possible for an agent to ignore the TDD procedure even when instructed to follow it, it can’t ignore a build failure.

View on HN

I didn't understand this part in the article. It seems advantageous to have an AI do manual mutation testing because it can actually introduce realistic bugs, like raising an exception or returning a different, known error code. Instead of the quite limited approach offered by automated frameworks that mostly just change binary operators. The whole issue with manual mutation testing is that it's slow and has to be done by hand, but with AI it's not by hand any longer and if it takes a few minutes in the background, who cares.

This surprised me too. I suspect the robots only got a "use mutation testing" prompt and independently decided that it must mean manual mutation testing, or possibly that they ran in a sandbox where an automated mutation testing tool was not installed.

But I'm surprised Dan Luu didn't make a remark about this. Surely he must know the difference!

He did remark on it: "Although mutation testing is a standard programming term, agents generally didn't actually do mutation testing and instead did normal testing with some small amount of mutating things in a way that isn't really mutation testing, similar to how the TDD instruction modified behavior but didn't get agents to do TDD."

If you're wondering why he doesn't go into greater detail, several testing methodologies earlier we have: "Since just saying that agents didn't really meaningfully do the thing is repetitive, I'll make these sections short and only highlight particular curiosities."

Yes, automated and gated. Zero missed rather than percentage. This morning harness failed the agent written test that passed on the fixed code and also passed against the mutated code. Test looked fine, code reviews would approve it, only the gate caught it! The agent didn't game it. Why I said zero missed, not a percentage? Because only one mutation survived and percentage threshold would probably swallow it.

Does anyone else gate at zero rather than a percentage threshold?

Do you have specific strategies on handling equivalent mutants?

I'm not generating the mutations automatically. Every one is a single targeted change assigned to a single test, reviewed one at a time. Thanks to that changes that mean the same thing don't stack up. The cost is reversed, I only catch what I thought about.

My real issue is different. This week one change removed the step which is creating a filename from the path and the test didnt catch it, it was passing. It wasn't an equivalent mutant, the test was looking at the wrong place. It works for me only because I'm working here on a single file, a complex bash script. Does anyone have a sensible way to limit equivalent mutants without manually checking every one that survived?

Would automated mutation testing actually make a big difference for agents, or would they just find new ways to game the metric?

I believe it would. Automated mutation testing is the test coverage metric that is nearly impossible to cheat.

Reply on news.ycombinator.com