Hacker News Reader

Comment by teacpde | Hacker News Reader

  The go test command now caches test results: if the test 
  executable and command line match a previous run and the 
  files and environment variables consulted by that run 
  have not changed either, go test will print the previous 
  test output, replacing the elapsed time with the string 
  “(cached).”

This is my favorite part of the release.

View on HN

I don't understand the utility. Why am I running tests again if nothing has changed? Is this just to speed up multipackage testing use cases?

I imagine if you're working on a project with 3 packages and only 1 package changed then the test will automatically skip the other 2 with no flags or changed command needed? Or perhaps if your working on one large package and only one file changed?

Yes that’s my understanding.

Atom's Go integration re-runs tests after save to a .go file. It's very convenient because you can see test failures and test coverage changes immediately.

But it's slow if you modified a file that is not involved in tested code paths.

This is just one concrete example. In general when the code size grows, very quickly it becomes impossible to tell if a given edit could possibly affect tests or not. So you want to run `go test` just in case so making it faster is a good thing.

But this isn’t doing ‘intelligent’ caching, just like go doesn’t do intelligent test checking. If the test binary changes (I think that implies basically any change in the package) the tests rerun.

In general the bar for making changes is: is it better that it was yesterday?

Before 1.10 `go test` used to re-run tests unconditionally.

Since 1.10 it'll avoid re-running them in some cases.

Neither me nor the announcement claims "intelligence".

The rules are simple and effective which is the Go Way.

While you could try to make even better, more "intelligent" rules for determining wether to re-run the tests, those would quickly run into engineering complexity chasing diminishing returns. That is not the Go Way.

There are lots of things on my list you could do that were simple & effective that would provide me more value than this.

Historically the argument against them have been about adding needless complexity. I wonder who was asking for this and how they got it prioritized.

It was requested by rsc in 2015: https://github.com/golang/go/issues/11193

It was implemented by rsc subsequently, after discussion on the issue: https://go-review.googlesource.com/c/go/+/75631

I'm not sure what you're trying to insinuate.

I’m not insinuating anything. I’m flat saying this seems like a low value feature. Other higher value features are denied on the basis of unneeded complexity.

I mean, the project lead wanted a feature and then implemented the feature, clearly it had value to him.

The difference is that this feature (ideally) only affects implementation complexity, not interface complexity. It can also be dropped again tomorrow without breaking the world.

This makes it wildly different from language changes, which require a lot more consideration.

Not a Go developer but bazel in Java does the same thing and it's really helpful when writing test cases. Say I am editing two test methods within a test suite of 30 test cases, it will run only those two tests. For the rest 28 it will show the cached output.

But thats the thing, this is not incremental compile/test. Its just checking the binary output (at least as far as I can tell).

That's different. Go will rerun all tests (for that binary)

Larger projects can now effectively be TDD'd in. Or rather you modify or fix a test in a test file and the tests compile and execute very fast. Or even simply the call to `go run`, `go build` and `go test` will be significantly faster.

This is the kind of difference I'm getting

GOCACHE=off time go build

       16.36 real        37.02 user         6.43 sys

GOCACHE=on time go build

        0.66 real         0.53 user         0.37 sys

Number of files including vendor dependencies (some files may not actually be imported)

find . -name '*.go' | wc -l

Those 2 runs are between 2 runs where something different change right? Thats effectively just outsourcing the 'did I change anything' memory to the cache system?

I'm ok with that I guess and I suppose I see the benefit when running multiple package test runs where their aren't dependencies between them. But I've just not seen the utility in practice, and it seems like it can introduce bugs where the cache is used incorrectly.

You are right, no changes. With a change, most objects are still valid as cached and can result in similar speedup. I'm also wary of caching introducing bugs and intend to continue to CI without it as a safety net. May even require a no cache build before pushing commits.

What does "files ... consulted by that run" mean? Does it track file/directory operations that occur during a test?

I'm a heavy user of table-driven tests that typically use YAML files that set up the parameters for the test scenarios. These use filepath.Glob(), ioutil.ReadDir(), and so on. For "go test" to invalidate its cache correctly, it would have to intercept those calls and record the dependencies. That'd be awesome.

Only source files, unfortunately. You could use a caching build system like Bazel to fork with data-driven tests, but most users don't need that functionality.

It would be nice if tests could give hints like t.DisableCache() or t.DependsOn(filename).

I think this is misinformation. go test does track files read during tests and records a hash of their contents.

Bug: https://github.com/golang/go/issues/22593 Commit: https://go-review.googlesource.com/c/go/+/81895

I looked at the code mentioned in the sibling comment. They do track file access.

Incredibly, they've modified the standard library (os.Exec(), Stat() and so on) to log the activity to an internal mechanism used by the testing package. I'm rather flabbergasted by the close coupling (it's a feature that would've been impossible for anyone outside the Go team to implement), though it's certainly a pragmatic solution.

Tracking seems to only occur within the current process, so if you call out to a subprocess to generate files, those files won't be taken into account, but I've not gone through the code with a fine comb.

In other words, there are ways to 'fool' the dependency tree. My first thought was that this might be a contrived situation. Then I realized I have exactly that in a project I'm presently working on. It does a lot of file system operations and I use shell scripts invoked from test code to set up files for testing. I'll have to investigate how to force tests to run if I change test scripts. I suppose changing the `go` test case to result in a test failure and then modifying the script to produce correct results (or a valid failure ;) ) would do the trick.

I guess changing the scripts would be detected by `go` so perhaps I don't have a problem. At present none of my scripts invoke other scripts.

Just read the scripts from your test function, then go will check their hashes and rerun the test if they changed.

Reply on news.ycombinator.com