Static reasoning can't prove runtime behavior: benchmark of AI PR reviewers | Hacker News Reader

Static reasoning can't prove runtime behavior: benchmark of AI PR reviewers

heal.dev

6 pointsbatikha4 comments

Hi HN, we're the team behind Heal. We recently wanted to measure the actual limits of static reasoning in LLMs: specifically, what classes of bugs AI code reviewers are fundamentally blind to because they lack a runtime.

To measure this, we ran a benchmark. We took 5 products (3 open-source, 2 private) and replayed 236 real pull requests from their history. We ran three popular AI PR reviewers on default settings, and then ran Heal, our agent that builds an application sandbox (to control the seeds, mock dependencies etc)around the application and drives it end-to-end like a user would.

A few insights:

- Stacking code reviewers doesn’t yield better results.

- A runtime allows LLMs to test and verify assumptions without risking increasing noise level.

- PR bots are still great for style, consistency, and files outside the user path (CI config, docs, etc.)

We stopped after spending a few thousand dollars running this because the signal was strong enough, but we’re hoping to grow the dataset. We also acknowledge that requiring a reproducible test (what Heal does) is a different standard than an LLM just leaving a comment.

We’d love to hear your thoughts on the methodology, or if you’ve run into the limits of static AI reviewers in your own workflows. Happy to answer any questions.

We recently replaced vendor solutions with a home grown one. It's doing so much better and is cheaper. The nicest thing is that we can iterate on our own, customizing it to our developer experience, feedback, and learning. For what you are offering, I imagine we would do the same (home grown) if we ever feel the need.

> We stopped after spending a few thousand dollars

How do you plan to charge for this? What are your thoughts on the SaaSpocolypse being likely to effect all the Ai startups? (ai tooling, not labs)

Good question. So far our users find value in having everything running smoothly with very low false positive and a continuously improving product.

(we are free during beta, then will charge at token cost for startups and small teams)

As for all products being replaced by labs / SaaSpocalypse: who knows? I think there's always value in making things extremely easy to use and token-efficient.

The thing is, Ai makes it easier to write a homegrown, custom alternative.

Its a real pain to deal with vendors, legal requirements, and feature requests. Vendors have to optimize for their average user, homegrown can be optimized for one.

Reply on news.ycombinator.com