The solution to running only e2e tests on affected files has been around long before LLM. This is a bandage on poor CI.
> Did Claude catch all the edge cases? Yes, and I'm not exaggerating. Claude never missed a relevant E2E test. But it tends to run more tests than needed, which is fine - better safe than sorry.
If you have some particular issue with the author's methodology, you should state that.
How many times did they conduct this experiment? Over how long time? How did they determine which tests were relevant and that Claude didn't miss them? Did they try it on more than one project?
My point was that none of this tells me this will work in general
We don't know the methodology, since the author does not state how he verified that function or how he would verify the function for a large code base.
There's a handy list to check against the article here: https://dmitriid.com/everything-around-llms-is-still-magical... starting at "For every description of how LLMs work or don't work we know only some, but not all of the following"
- Do we know which projects people work on?
It's pretty easy to discover that OP works on https://livox.com.br/en/, a tool that uses AI to let people with disabilities speak. That sounds like a reasonable project to me.
- Do we know which codebases (greenfield, mature, proprietary etc.) people work on
The e2e tests took 2 hours to run and the website quotes ~40M words. That is not greenfield.
- Do we know the level of expertise the people have?
It seems like they work on nontrivial production apps.
- How much additional work did they have reviewing, fixing, deploying, finishing etc.?
The article says very little.
And that's the crux, isn't it. Because that checklist really is just the tip of the iceberg.
Some people have completely opposite experiences: https://news.ycombinator.com/item?id=45152139
Others question the validity of the approach entirely: https://news.ycombinator.com/item?id=45152668
Oh, don't get me wrong: I like the idea. I would trust LLMs with this idea about as far as I could throw them.
Running all E2E tests in a pipeline isn't feasible due to time constraints (takes hours). Most companies just run these tests nightly (and we still do). Which means we would still catch any issues that slip through the initial screening. But so far, nothing did.
This doesn't work in distributed systems, since changing the behavior of one file that's compiled in one binary can cause a downstream issue in a separate binary that sends a network call to the first. e.g. A programmer makes a behavioral change to binary #1 that falls within defined behavior, but encounters Hyrum's Law because of a valid behavior of binary #2.
- avoid distributed systems at all costs
- if you can't avoid them, never make breaking API changes
Risks: missing e2e tests that should have run letting bugs into production, more time spent chasing down flakes due to non determinism.
Benefits: increased productivity, catch bugs sooner (since you can run e2e tests more often).