Hacker News Reader

One metric I'd be interested in is defects discovered per hour of test maintenance. Traditional end-to-end tests can become expensive when interfaces change frequently.

For AI-driven testing, I'd want to compare:

1. Real defects discovered before release 2. False positives requiring investigation 3. Maintenance effort after UI changes 4. Bugs that still escape into production

Have you benchmarked these against conventional end-to-end tests on the same application?

View on HN

Hey there! Definitely good points here. I think some of these are covered in the benchmark, for example, false positives requiring investigation is equivalent to (1-precision), bugs that still escape into production is (1-recall), and real defects is recall.

Our goal with Mo is to make maintenance effort as low as possible while still maintaining quality. One thing we've been talking about separately is running a benchmark where all of the agents have virtually unlimited time (or something really high like 24h) and they need to find as many bugs as possible. Is that more similar to what you're looking for?

Reply on news.ycombinator.com