Predicting Test Failures

We released Ekstazi tool (http://ekstazi.org/) last year that implements novel test selection technique for Java. (The details of the technique are discussed elsewhere.) Ekstazi has been adopted by several projects. Hopefully it will be useful for some developers that see these lines.

I love this idea, because my developers could just run the tests that were most likely to have changed locally, and then we can run the full test suite (in parallel) on our CI machines. Of course, to reach the full potential, the coverage output (from the full run) would need to be added to the project to determine what individual tests should be run.

One thing to suggest:

Instead of picking what tests to run, it may be better to change the order in which tests are run instead.

In particular: sorting tests by "how often did this test change results when these lines were modified?", descending.

(And also running a background job that asserts that tests don't change results when run in a different order.)

Greg Young did the same thing in .Net; he built a product out of it. In the end he figured out it wasn't a viable business, so he made it open source. You can download it here http://www.continuoustests.com/ and the code is OSS.

There's a pretty well known plugin that does something similar to this in Java land; https://infinitest.github.io/. Integrates with both IntelliJ and Eclipse. Works wonders.

Had the exact same idea. Never finished it. He's on to something here. Take note.

This will still force you to run most of the test suite if you make changes to a "god object"

I think HP's Application Lifecycle Intelligence (ALI) tool does something like this. Yes, it's enterprise software but worth a look at their approach.

"LOGS = []" this is bad ruby, shouldn't use global constants neither as a variable or as mutable.

      end
    end
  end

end