That's a great point!

Depends on the what you define by "externality".

I like to look at it this way, what's being tested? I see three groups:

  - testing the software you wrote.
  - testing integration with other software you wrote (microservice interaction maybe?)
  - testing integration to third party software.

I'm telling you our current plan, feel free to poke holes:

In the first case the only externalities that seem to really matter are mostly resource and configuration files. We could detect I/O with such files and rerun the test if the underlying resource/config changed.

In the second case were planning on collecting coverage for all services being tested and then employ a more sophisticated diff heurastic.

For the third case mark those tests as always run, hopefully you are testing your codebase and not just thirdparty integrations.

We have a poc of a syscall tracer that gives us an I/O map of the test being run or the software being tested. this includes reading/writing to files, pipes, network etc...