daniiel

The screenshot script is the scarier half of that story, because the seven failing tests did their job while the previews kept coming out at 40% and looked like work getting done. I had a similar situation, the engine landed on a screen that looked almost like the expected end state and reported a pass, and it took a person going through the step screenshots by hand to notice. Did measuring inside the frame fix it for good, or you need to check the previews as well?

Not sure precision/recall is the number I'd look at first, because the thing I'd actually want to know is how often the tool says pass on a flow that is already broken, and a silent pass is worse than a flaky test, since the flaky one at least makes some noise.

I'm building something in the same space and this bit me quite early, so what I ended up doing was basically mutation testing: take a handful of open source apps, plant bugs on purpose, and count how many of them the tool waves through anyway. Seems that the pass/fail counts were not the interesting part at all, the interesting part was letting the engine answer with a third option, "I can't tell", because once that was allowed the false passes mostly disappeared, it simply stopped guessing. I would say a guessed pass is worse than having no test, and I keep every run replayable for the same reason, otherwise I wouldn't trust my own numbers either.

Curious how Mo (nice name) handles the middle ground - the docs say a bug it can't reproduce is not reported as confirmed, but does it show up anywhere or is it just dropped? And for QA Bench, did you plant bugs yourselves or only label existing ones?