batikha

Good question. So far our users find value in having everything running smoothly with very low false positive and a continuously improving product.

(we are free during beta, then will charge at token cost for startups and small teams)

As for all products being replaced by labs / SaaSpocalypse: who knows? I think there's always value in making things extremely easy to use and token-efficient.

Hi HN, we're the team behind Heal. We recently wanted to measure the actual limits of static reasoning in LLMs: specifically, what classes of bugs AI code reviewers are fundamentally blind to because they lack a runtime.

To measure this, we ran a benchmark. We took 5 products (3 open-source, 2 private) and replayed 236 real pull requests from their history. We ran three popular AI PR reviewers on default settings, and then ran Heal, our agent that builds an application sandbox (to control the seeds, mock dependencies etc)around the application and drives it end-to-end like a user would.

A few insights:

- Stacking code reviewers doesn’t yield better results.

- A runtime allows LLMs to test and verify assumptions without risking increasing noise level.

- PR bots are still great for style, consistency, and files outside the user path (CI config, docs, etc.)

We stopped after spending a few thousand dollars running this because the signal was strong enough, but we’re hoping to grow the dataset. We also acknowledge that requiring a reproducible test (what Heal does) is a different standard than an LLM just leaving a comment.

We’d love to hear your thoughts on the methodology, or if you’ve run into the limits of static AI reviewers in your own workflows. Happy to answer any questions.

Very cool! I already can see a lot of "this is already solved by playwright/cypress/selenium/deterministic stuff" in the comments.

Over nearly 10 years in startups (big and small), I've been consistently surprised by how much I hear that "testing has been solved", yet I see very little automation in place and PMs/QAs/devs and sometimes CEOs and VPs doing lots of manual QA. And not only on new features (which is a good thing), also on happy path / core features (arguably a waste of time to test things over and over again).

More than once I worked for a company that was against having a manual QA team, out of principle and more or less valid reasons (we use a typed language so less bug, engineers are empowered, etc etc), but ended up hiring external consultants to handle QA after a big quality incident.

The amount of mismatch between theory and practice in this field is impressive.

I work in the field and built a tool that has way less flakiness than deterministic solutions. The issue is testing environments are always imperfect because (a) they are stateful and (b) there's always some randomness in actual production software. Some teams have very clean testing environment but most don't.

So being non-deterministic is actually an advantage, in practice.

Jevon's paradox though? Planes are so much more efficient now than the first prototypes, yet usage is so much higher that resource consumption due to airplanes have vastly increased. Same goes with generative models.