momentichenry

Nice! Let us know if we can help. Just to save you some time, I will say that in order to speed up data labelling, we pinned a commit a little bit behind latest main, since some of the bugs would already be documented in fixes or GitHub issues. So some of the bugs may have already been fixed!

Hey there! It's not strictly find a bug, as it's also getting the skill for the respective browser tools, but this is fair criticism. I think the current bench shows that if you don't have the time and money to build you're own harness, you're much better off using Mo. However, we do want to start including some of the harnesses that customers/users built before switching to Mo for comparison.

Have you built your own? Would love to see your results and add them to our site!

Thank you!

Currently it's only simulators and emulators, but we'll be rolling out real devices in the next few weeks.

Yes! We create a knowledge base entry so that Mo learns from its mistakes. This is especially helpful for stuff that's actually intended behavior, because at the end of the day, some things just come down to taste.

Hey there! This is coming soon! Although I will say, when we did an earlier run of this, Codex and Claude actually figured out how to look up the source code for the fixture in the public GitHub repo, and while it improved recall a bit, it also caused them to hallucinate more issues due to assuming that their reading of the source code was correct.

Mo is mostly looking for user-facing failures, or at least things that impact the user in some way. So if the app was super slow due to this, or was timing out, it would likely find it. However, if this is purely a backend cost issue, it currently wouldn't be likely to flag it.

Mo is able to inspect the network traffic, although I'm curious how many bugs you see that truly have zero observable impact, and if they do indeed have no impact, do they matter?

That's fair, and I think also something we've seen while creating this benchmark: it's incredibly difficult, if not impossible, to annotate every single bug in an app. This is actually sort of what we were trying to capture with severity-weighted score: the agent should get way more credit if it finds really bad bugs than if it finds a bunch of minor UI quirks, although this could definitely be improved upon when we release a v2 of the benchmark.

Hey there! Definitely good points here. I think some of these are covered in the benchmark, for example, false positives requiring investigation is equivalent to (1-precision), bugs that still escape into production is (1-recall), and real defects is recall.

Our goal with Mo is to make maintenance effort as low as possible while still maintaining quality. One thing we've been talking about separately is running a benchmark where all of the agents have virtually unlimited time (or something really high like 24h) and they need to find as many bugs as possible. Is that more similar to what you're looking for?

Hey there! Yes, using different models for performing the work and verifying is definitely something we've run into as well. In fact, within Mo we use one model for finding suspect bugs and a different one for checking its work to avoid this exact failure mode.

Mo doesn't need specs to run, but it does support PR review-mode, or just open ended exploration. If you do have specs you want it to execute, you can pass those to the agent as well.

Hey there! Thanks for watching our demo! Would love to run Mo on some other, public, benchmarks and share results. I'm curious how you're evaluating your framework's ability to find bugs vs just execute flows in an app? I also took a look at your site and it seems like you're product is more akin to Momentic regression tests, as opposed to Mo, as it depends on pre-written test cases rather than exploring itself. Maybe a better comparison would be our AI action?

I think it's true that our biggest gains vs other agents are in codebase-wide sweeps, because to your point, PR checks are limited in scope. However, there's still value in running Mo on those features because:

1. It should be pretty efficient using prior knowledge from other runs.

2. It helps expand the graph as your app evolves so that the DAG is always up to date.

Thanks for the feedback.

- Your first point sounds like a frontend bug to me. I wonder if there's missing revalidation.

- Sounds like the defaults need a little more tuning. You can actually update the definition yourself in settings https://app.momentic.ai/settings/mo.

- I'll take a look at this "stop testing" button thing as well. Thanks for flagging.

- What kind of export would you like? We currently have integrations that allow syncing bugs to Linear or other project management software. You can also export the full results using our CLI: `qa report <session-id> --output mo-report`.

Hey there! I'm not 100% sure how many came from Mo originally, but I can add a bit about our process:

- The candidate ground truth bugs are generated using runs from all agents

- Every candidate gets reviewed by the other agents to generate a recommendation (just makes human review easier)

- A human reviewer makes the final call (no candidates are discarded just because Mo couldn't reproduce)

And in terms of grading the final runs:

- A lot of the final runs still found a small percentage of novel bugs (5-10%), because it's extremely difficult to build a comprehensive list of all bugs, even after hundreds of runs.

- Nothing that didn't match was auto-discarded (read - it wasn't rejected because Mo couldn't repro).

- All unmatched findings were run through the same human review process as above.

- All the runs were regraded after human review to get a final score with zero unmatched findings.

One interesting thing we did notice during this is that Codex and Claude tend to converge on the same findings run over run (very few initially unmatched findings on final runs), whereas Mo tends to find a lot more novel bugs on ever run.

Hey there! This is Henry, one of the engineers working on Mo. Thanks for reading our post! A lot of interesting stuff here:

- Precision/recall vs pass/fail: I really like this point, and I would say QA bench is actually not a great fit for measuring pass/fail currently, because all of the fixtures have numerous bugs in them, and almost all agent runs find at least one (so the fail rate is basically zero). My assumption would be that recall is correlated with pass/fail accuracy, but we are working on a PR-review variant of QA-bench that I think would be more suited to measuring pass/fail directly.

- I think the insight about allowing "unsure" is interesting, and this is actually similar to what we have for some of the completions in regular Momentic tests, although I'm curious how you generate something useful for the user from the agent basically saying "there might be a bug in this code but I don't know what it is."

- Internally, if Mo can't reproduce a suspected bug it gets discarded. Although we do review some of these to make sure that there aren't issues with the reproduction agent.

- QA bench has zero planted bugs; they're all from the original source. Initially, we did try planting bugs and even generating apps with given bugs using an LLM, but we found that this didn't actually help us establish the ground truth because the agents were still finding bugs that we hadn't planted and needed manual review.

This sounds a lot like what we call an "AI check" in a regular Momentic test. These work great on known flows, and are actually one of the main reasons a lot of our customers love Momentic. Mo doesn't replace that; it builds on top of it by adding the ability to discover and test new flows that you didn't previously write scripts for, or don't want persistent scripts for. In the near future, Mo will also be able to propose durable tests that you can rerun later (essentially, exactly what you're describing), which will decrease cost and increase consistency significantly for already known flows.

Hey there! Sorry to hear that the onboarding experience didn't meet your expectations. Would you mind reaching out to support@momentic.ai and sharing your account details so that we can debug this? We'd love to help you get set up and start using Momentic.

Hey! I'm Henry, one of the engineers working on Mo. To answer your questions:

- I expect there will be many more competitors to come, as there already are. I also think achieving these performance improvements is non-trivial and depends on a lot of internal tooling we've built over the last two years (see next answer). As with any software, there will obviously be new competitors that match our historical performance eventually, but we will also be continuing to evolve our product. As you can see from the recall numbers in the benchmark, there is still a lot of room for improvement.

- Mo is really our second product. Over the last two years we've been building the Momentic e2e test framework and tooling around it, which allows engineering teams to own quality by authoring tests in plain-english and maintaining them with little to no effort. Mo builds on top of that nicely because we've already spent a lot of time tuning the underlying tools so that agents can use them efficiently to get better results.

- Real device support is already in the works! It should go live in the next couple of weeks. Would love to connect once it's ready.

Update: we just added iOS 27 support. You can select it in the session settings dropdown. Feel free to try it out and let us know what you think!

Thanks! That's fair feedback. We'll update the writeup to make this clearer. We're also hoping to release a v2 benchmark soon with more fixtures and hopefully an even more optimized Mo.

Hey! I'm Henry, one of the engineers working on Mo. Currently, Mo is super focused on being the absolute best at finding bugs, but isn't meant to be a general purpose coding agent. For that we ship a CLI (https://github.com/momentic-ai/qa) and matching skill which you can use to integrate Mo with whatever coding agent you prefer. One pretty successful setup we've seen a lot of customers use is telling the agent to build a feature, then use Mo to test it, and only go back to the user for input once all the bugs are resolved.

Hey! I'm Henry, one of the engineers working on Mo. You're looking at the right numbers, but the takeaway is slightly different from ours.

The main thing we're trying to show is that Mo finds over 3x as many correct bugs in the same amount of time as the next-best QA agent. This is the main thing we optimize for since the worst thing you can do in QA is miss a real bug and ship broken code to prod. Currently, the tradeoff you make for that better recall is higher cost. I would say the precision is more or less comparable, but marginally lower.

Obviously we'd love to be beating every other agent on all of these numbers, and we're actively working on it, but for now this is an honest and transparent look at how the tool is doing.

Feel free to take a look at the full writeup for more detail!

Hey! I'm Henry, one of the engineers working on Mo. The approach to testing itself is very similar to UAT, except that it's fully automated. This means you can speed up the feedback loop significantly. Imagine instead of having to wait for the next release and run a time-consuming manual bugbash in order to find out that your changes broke something, you can get feedback immediately on any PR you want, at a fraction of the cost.