Hacker News Reader

Comment by MGabrielDenton | Hacker News Reader

Hey I'm Gabe (one of the engineers @Momentic). In general Mo's advantage is in the total amount of bugs found. In comparison to claude code we find roughly 5x as many bugs at 35% less per bug (for the 60 min session).

View on HN

Thoughts on the bitter lesson? Eg what if models just keep getting better?

Good question, in short our harness & e2e tests act as compliments with the frontier rather than competitors. Architectural choices and data you provide really matter for domain specific tasks. Mo will be working with harness specific tooling like: previous momentic test results, our KB, and eventually creating new Momentic regression. Having smarter/more capable models will improve the efficiency of AI generated tests (coverage/runtime) and ideally do less repetition in exploratory testing.

What's really being asked is, what if you stop performing any better than an unmodified frontier model with no system prompt?

It seems you are betting on frontier models improvement being slower than the pace of new features in your tool

If this bet doesn't pan out, the only thing I can think of is pivoting into something else

Then stock claude code??? that's like... the lowest of bars.

You're right stock CC was pretty terrible. Thats why we compared against CC && Codex with agent browser and playwright mcp as well as devin. Do you have any other agents / configurations you'd like to see?

We had someone recommend browser-use and are running it against the benchmark currently. We're hosting the results here if you'd like to see more: https://momentic.ai/qa-bench

I think all of those are very naive approaches. What I currently do is script out screenshot + application state pipelines and then run those screenshots and states through clef with questionnaires. The scripts are deterministic, clef costs pennies or less and is also reliable, the questionnaires are derived from the script with escalation to stronger lmm's if necessary (historically very infrequent from my usage). It isn't a "drop in" solution, but it is fairly trivial to set up and you can attach it to CI. It isn't perfect, but I've found it to be more successful and way more cost effective than throwing Claude or GPT models at it regardless of harness.

This sounds a lot like what we call an "AI check" in a regular Momentic test. These work great on known flows, and are actually one of the main reasons a lot of our customers love Momentic. Mo doesn't replace that; it builds on top of it by adding the ability to discover and test new flows that you didn't previously write scripts for, or don't want persistent scripts for. In the near future, Mo will also be able to propose durable tests that you can rerun later (essentially, exactly what you're describing), which will decrease cost and increase consistency significantly for already known flows.

Gotcha, so it's a harness optimizing narrowly for cost per task in discovering QA testing flows and eventually documenting those flows durably? Anything stopping someone from asking Mo what their context and tools look like?

Reply on news.ycombinator.com