_cfl0

That's a great point!

Depends on the what you define by "externality".

I like to look at it this way, what's being tested? I see three groups:

  - testing the software you wrote.
  - testing integration with other software you wrote (microservice interaction maybe?)
  - testing integration to third party software.

I'm telling you our current plan, feel free to poke holes:

In the first case the only externalities that seem to really matter are mostly resource and configuration files. We could detect I/O with such files and rerun the test if the underlying resource/config changed.

In the second case were planning on collecting coverage for all services being tested and then employ a more sophisticated diff heurastic.

For the third case mark those tests as always run, hopefully you are testing your codebase and not just thirdparty integrations.

We have a poc of a syscall tracer that gives us an I/O map of the test being run or the software being tested. this includes reading/writing to files, pipes, network etc...

Meta and Google employ AI based methods for skipping tests (See):

they are pretty much flawed in a way, because they require you to wait weeks/months to gain statistical sagnificance. And even then they are too costly to train to make sense in smaller scales.

There's an ex-Google employee who came out with something similar in 2005 for JUnit java tests, since erased it was called google-testar.

We talked to him and ulitmatly he sunsetted the project because:

1) static analysis back then was basically copying language parser code from the Language distro and retro fitting it which was HARD to maintain.

2) Misses when resource files were changed, he was a one man show to my understanding.

Our View: Static analysis has gone a long way since... and yet no one has made something for system tests and integration tests, there's a couple unit tests solution out there.

See: https://static.googleusercontent.com/media/research.google.c...

https://research.facebook.com/publications/predictive-test-s...

How it works: First time you run nabaz:

1. Collects code coverage for every tests that runs, seperatly.

2. Saves code coverage in local sqlite db, along with commit ID as key.

Second to n runs later:

1. Traverse git tree and queries db via commit ID -> per test code coverage

2.a git diff between new code (code change) and the code coverage from the old commit, test case by test case.

2.b. If an intersection between new code and old code coverage is found, test is rerun. else, it will be skipped.