How do you deal with tests that interact with something external to the codebase? The result of those could potentially change without a code change.
Depends on the what you define by "externality".
I like to look at it this way, what's being tested? I see three groups:
- testing the software you wrote.
- testing integration with other software you wrote (microservice interaction maybe?)
- testing integration to third party software.
In the first case the only externalities that seem to really matter are mostly resource and configuration files. We could detect I/O with such files and rerun the test if the underlying resource/config changed.
In the second case were planning on collecting coverage for all services being tested and then employ a more sophisticated diff heurastic.
For the third case mark those tests as always run, hopefully you are testing your codebase and not just thirdparty integrations.
We have a poc of a syscall tracer that gives us an I/O map of the test being run or the software being tested. this includes reading/writing to files, pipes, network etc...
What are the odds this baby will be made compatible with Java?
However they are usually testing components from end-to-end. This seems not very amenable to benefitting this approach, since "side effects" are an inherent part of nearly every integration test.
Regarding side effects, completely fine as long as they're mapped out. Would love to hear more, nothing pitchy or salesy.