Hacker News Reader

Comment by mabbo | Hacker News Reader

One (misguided) counter argument I've heard from otherwise fantastic devs it's the notion of adding randomness to unit tests in the hopes that if there's a bug, at least some builds will fail. In practice, I've seen those builds and developers saying "yeah, sometimes you need to build it twice".

I think the solution is to give those devs who favor such techniques a separate but easy to use fuzzing tool set that they can run just like their unit tests, separate from their usual 'build' command. Give them their ability to discover new bugs, but make it separate from the real build.

View on HN

Why would the randomness in unit tests affect the binary? RNGs are invoked when the tests are run, not when they're built, and anyway, test code shouldn't be part of the final binary.

The test suite is used to validate the build. Intermittently failing test suite -> intermittently failing build. You can always disable running a problematic test suite, but that doesn't exactly inspire confidence in the result.

My take on that is that those devs are probably fooling themselves about the state of their test suite. I'm working on a codebase like that right now - it's in generally a relatively good state, but just this week I've run into a moderately nasty bug involving global state not getting properly reset which the tests didn't cover because their specific execution order happened to work.

Or just seed the PRNG with some digest of the relevant source code (perhaps as simple as the version number). Doesn’t solve the problem that your tests can suddenly break on unrelated changes, but does solve the problem that your tests can break without any changes.

Down-voted because that's not an argument against reproducible builds. The outputs can be bit-for-bit identical but still use an RNG for fuzz testing.

A "build" often includes running the unit test suite to verify that the software works in the build environment, so randomness in the test suite can indeed lead to non-reproducible output: when a bad RNG seed causes a test failure and thus a failed build. I've spent my fair share of time debugging issues like this.

Use a PRNG seeded with the hash of the build manifest.

I don't know what a build manifest is but yeah, that sounds great. However, the test suite needs to support this. Plenty of times they do not.

Non-determinism should be on a leaf of the DAG.

Reply on news.ycombinator.com