Hacker News Reader

Comment by daveguy | Hacker News Reader

I like the concept and it seems that "run them all and see what changed" is a great model to avoid missing any side effects from a code change. However, if you're running all the tests on every change, doesn't that mean the responsiveness is necessarily greater than 250ms if your test suite is longer than 250ms?

Regardless, running tests in the background to eliminate human delay is good. It would be nice if the (~test time)*(test change likelihood) of the tests were used to order tests based on fastest relevant feedback.

A code->test change likelihood map may also be used to reveal previously unexpected dependencies. Also ~test time could be difficult (see halting problem and windows time remaining estimates).

This is good work.

View on HN

This tool doesn't run all the tests, only the ones affected by the code changes since the last run. It figures out which tests to run by initially running all tests, and storing the code coverage of each test.

It doesn't need to rely on "test change likelihood" - if a code change is outside of the code coverage of the test, it doesn't affect the test.

So...like gradle?

> If a code change is outside of the code coverage of the test, it doesn't affect the test.

This assertion questionable, the extent it is true it is language specific.

Here is a counterexample in Java:

  // foo.java 
  public class Foo {
    public static final int LENGTH = 10;
  }

  // bar.java 
  public class Bar {
    public int getFooLength() {
      return Foo.LENGTH;
    }
  }

Code coverage tools will not highlight Foo as being touched when Bar.getFooLength is invoked, even though a test asserting that getFooLength() is 10 will fail or succeed depending on Foo.LENGTH. The reason for this is that Foo.LENGTH is inlined in Bar. If you for example wanted to patch a new version of the value, you need to supply Bar.class to affect the change; patching Foo alone would do nothing.

C and C++ likewise do not satisfy this requirement because macros.

You'd want to augment the code analysis with the dependency graph of object/class files (or use it instead of what the OP is using). I'm not sure if it's bullet-proof, but except for runtime dynamic linking, you should get a superset of all affected changes if you just use... whatever your IDE uses to determine which files need to be rebuild when you make a change. Following that graph should give you a superset of all unit tests that need to be run.

Like, e.g. if I change a macro in a header file and press "rebuild" on the project, MSVC (or MSBuild driven by CMake) will figure out that the header was changed, chase down which translation units include it directly or transitively, and rebuild those, then link the output and... chase down everyone else who links to the output and relink them, etc.

I bet you could produce a counterexample that breaks this mechanism (C++ being what it is), but I don't expect to see it in an actual codebase.

Not bulletproof, but a-ok if your running a full run just before the commit.

The ordering feature is actually being built as we speak. analyzing liklihood can still be beneficial to cut down time even more.

If I would get a break from hn comments I could finish it today lol ;)

Reply on news.ycombinator.com