Hacker News Reader

Comment by chriswarbo | Hacker News Reader

> engineers don’t get paid to write tests, they get paid to produce software

I've heard this double-standard before, and it sounds plausible at first glance, but is quite demonstrably nonsense.

1) Automated tests are software

2) Engineers aren't paid to produce software (nobody would pay me to auto-generate a billion "hello world" programs per second). Engineers are paid to solve problems and/or create value. Tests solve problems (finding bugs, demonstrate usage, clarify thinking, document intent, etc.).

3) Software is a liability, not an asset. We should produce as little as needed, and delete it if possible.

4) Engineers aren't paid to perform manual testing (running commands over and over, clicking around web forms to see what happens, etc.)

View on HN

> 1) Automated tests are software

Which is assumed to be free of defects because people are suckers for circular reasoning.

What are you talking about? The whole point of automated testing is to confront the empirical fact that all software tends to have bugs, so we shouldn't have much confidence that we got it right.

We can increase our confidence by trying a few examples, but only a little, since the scope for failure is so huge.

Hammering a system with thousands of edge-cases, interleavings, previous regressions, etc. can give us a bit more confidence; but takes a huge amount of time for a person to do manually. It's pretty easy to automate though, and there are a load of libraries and frameworks that make it even easier.

If we want even more confidence in a system, we can verify it against a machine-checkable specification. That's still too hard to automate in most cases.

In any particular situation we need to find an appropriate balance between the cost of increasing our confidence in the system (i.e. how much effort we're willing to spend on prove ourselves wrong) versus the cost of failure.

Automated tests (especially property checks) are so low-cost and high-yield that I find them always worth doing, unless the consequence of failure is near-zero.

What logically follows from the two statements below:

1. "Automated tests are software" 2. "all software tends to have bugs"

It follows that automated tests tend to have bugs. So what? That doesn't at all imply that we shouldn't use them.

You're presumably using a whole heap of software to post your comments, all of which will tend to have bugs; yet you still find it useful.

Boolean logic rarely captures the nuances of real-world situations, and the "rounding errors" accumulate at each step of such arguments, which can lead to some silly (but logically consistent) outcomes. I think you've fallen victim to this here.

In a similar way, it's a false dichotomy to think of software as being either buggy xor perfect; of testing (or any other QA) as proving brokenness xor correctness; or to consider bugginess as implying uselessness.

Notice that I framed things in terms of confidence (in a Bayesian sense) and costs; not absolutes.

If you want true confidence in your functional tests then yep, [you need to write tests for them](https://www.jvt.me/posts/2018/11/07/unit-test-functional-tes...) - may seem like tests all the way down, but huge benefits reaped in the team + projects I'm on

I find #4, exploratory testing far more bang for buck than automated testing. I would argue engineers are paid to make services (software is a service) and exploratory testing is far more valuable towards that goal than automated tests.

a) users do crazy things. b) eating your own dogfood is worth it.

If by "exploratory testing" you mean things like clicking around on a site manually, then sure it can be useful to spend a couple of minutes looking for anything dodgy.

That doesn't mean engineers are paid to go through a list of regressions and edge-cases, look for broken links, double-check calculations with pen-and-paper, etc. All of that can be done across an entire site automatically in seconds; there's no excuse not to.

> a) users do crazy things

Again, thinking we can check such things better than a computer is misguided. Test frameworks like QuickCheck can stress-test a system with far more strange inputs/transitions than I could hope to think of.

Totally agree, any repeatable action can/should be automated. It hurts to see software engineers clicking login to test it still works.... Every day

If it's repeatable it isn't exploratory testing. The point of exploratory testing is to explore, not repeat ovee and over.

QuickCheck is a wonderful tool. Unfortunately most engineers go and write manual unit tests. But even with QuickCheck it is dumb to to test at the function level. Better to blackbox fuzz test.

I am dumb -- I test at the function level. I have a theory that the likelihood of catching a bug depends on the number of lines being tested, if each can be assumed to do its job. This implies that if a line calls a function that I wrote, I need to test that function. There is some work based on Kolmogorov complexity that seems to support this but didn't go very far (I did not see any follow up papers). Maybe it's all wrong. I am biased: I wrote quickcheck for R. (by the way, fuzz testing and randomized testing are two separate things).

Oh I am so tired of clicking around web forms to see what happens...

Seems most of my days are made of this lately.

Tests are a great way to take away tedious, boring, uninteresting essential tasks that can and will wear down a team.

Reply on news.ycombinator.com