Hacker News Reader

The question that always comes to mind whenever testing is discussed is "how do you know the test code is itself free of bugs?"

I distinctively remember once posing that question in a meeting about testing, and a manager replying --- seriously --- with "then perhaps the test code should itself have tests." Someone else must've come up with that before too, because (at a different job) I've also worked on a codebase where a surprising number of tests were basically testing the function of another test.

You don't know that the test code is free of bugs. You don't need to.

Case 0: No bugs in the test code. All is well.

Case 1: Bug in the test code that causes some bugs in the real code not to get caught. That's bad, but you're no worse off than if you didn't have the test at all.

Case 2: Bug in the test code that causes correct real code to look buggy. Result: the test fails, you look for problems, most likely you find that the problem is in the test code and fix it. Going forward, you have a working test.

Case 3: Bug in the test code that makes something else break. This can happen and is genuinely bad, but (1) it only affects testing, not your actual product, and (2) most bugs don't behave that way.

The test code is a net win if the bugs it catches in your real code are worth the effort of writing and debugging the test code. That's no less true on account of the possibility of bugs in the test code. It just means that when you estimate the benefit you have to be aware that sometimes the tests might be less effective because of bugs, and when you estimate the cost you have to be aware that you have to debug the code, not just write it the first time. And then you just ... decide what's the best tradeoff, just like everything else in engineering.

(And no, you don't need tests for your test code. The test code is a test for itself as well as for the code it's testing.)

I see 5 cases:

    Code  | Test  | Result
    --------------------------------------------------
    fine  | fine  | a) We have a regression test, yay!
    fine  | buggy | b) Someone breaks the code to make 
                       the test work. Oops.
                    c) Someone fixes the test, we have 
                       a regression test, yay!
    buggy | fine  | d) We'll fix the code, and we have 
                       a regression test, yay!
    buggy | buggy | e) Bug remains in code. Oops.

If the probability of introducing an error is p,

    Code  | Test  | Probabilities      | p = .01 (one in ten)
    --------------------------------------------------
    fine  | fine  | a) (1-p) * (1-p)   | .9801
    fine  | buggy | b) (1-p) * p * .5  | .00495
                    c) (1-p) * p * .5  | .00495
    buggy | fine  | d) p * (1-p)       | .0099
    buggy | buggy | e) p * p           | .0001

So we see, probability for:

    no harm done                 .9801  (a)
    bugs found + fixed           .01485 (c,d)
    bugs introduced / not found  .00505 (b,e)

The above completely ignores the fact, that depending on the code base there will be significantly more test code than production code. But then test code is quite often highly redundant, and might actually have a lower defect rate itself.

Also the probability on introducing an error in the production code and the test code, might actually not be statistically independent, which I assumes here. So take with a grain of salt.

[Edit] Actually d) could also end negatively. Guess a working model would have to take into account that on failing test cases, a sensible developer should take a step back and reason about why this happened. So the negative outcomes would be (hopefully) less likely than the positive ones here.

Top notch explanation here. P = 0.01 is 1 in 100 though right?

Oops, yes.

I must have missed some steps, but why did you assign p = 0.01, and how did you arrive that a) get to be (1-p) * (1-p)? And why can't, say, a) be p * p?

p is the probability of introducing in error into code.

To get some actual numbers, I choose an error rate of 1 in 100 lines of code (0.01). This is totally subjective and probably larger then in reality, but it does not hurt to be pessimistic here.

a) => the production code, nor the test code contains any bug. So if the probability for an error is `p` (.01), the probability for error free code `1 - p` (.99). If both events are independent, we can multiply them to get the probability of both happening at once `(1 - p) * (1 - p)`.

Add to this countless hours going into engineering tests that have to bent over backwards and mock the universe itself to test a tiny bit of logic that doesn't make sense to test in the first place. Then multiply with the hours getting lost trying to debug frail tests that break time-expensive continuous integration pipelines.

Good unit and integration tests are a rarity. Instead tests like the above ones which actually are being developed like a side-project to the real project are the norm and they are bogging down the whole project. But you cannot deliver any code that is not "covered" because that would be against the current 100%-TDD bible/best practice/call it however you want.

So the next dev/maintainer has to work out a badly written test (probably bearing untrue assumptions about the programs intended business logic) and after some fun hair pulling he does the reasonable thing which is working around the test or pampering the test to get it to pass. And that is how you end up with test code that is actually buggy and problematic in itself and does not really test much but it is increasing the test-coverage holly counter.

What the author didn't consider is that test code adds redundancy to the code base - if you introduce a bug only in the test code, while the tested code is correct, will make the test fail, and you will fix this.

Where instead the amount of test code can become a problem, in my experience, is with maintenance. Efficiency is very important.

You can tune the pixel difference in Selenium testing so it doesn't fail when under the acceptable pixel change limit. If it's off by one or two no biggie but off by 30 or 50 then maybe there was a structural change to the page, so it should fail. Something like "misMatchTolerance" from wdio-visual-regression-service[0] would allow for tweaking this. I'm sure there are similar tools for other languages.

If the UI tests catch bugs during development or help the team during a data migration, they're probably still worth having.

[0] https://github.com/zinserjan/wdio-visual-regression-service

Maybe the spry takeaway is that you should write tests for things you want to make sure they will not break.

Your UI being off by a pixel won't break your application, so if a test hangs on that, then it is not a good test.

However, your business logic, or network protocol routine, those should not break even if you heavily refactor or add new features (especially business logic where a broken behaviour might seem correct), so those need to be heavily tested.

If it is hard to test the juicy parts like business logic without also dragging in the UI, different OS/platform/db parts, etc, then you should look at how your application is structured and if it is really optimized for writing good tests.

>Your UI being off by a pixel won't break your application, so if a test hangs on that, then it is not a good test.

For brevity in the previous comment, I didn't fully flesh out the background on why fragile UI tests get created. It happens accidentally.

What sometimes happens is the the UI tester uses a "macro recorder" to record mouse movements and clicks. But then, a programmer shifts the position of a zipcode field by one pixel which throws the script off because it expected UI elements in a different spot. Fixing the broken UI tests is time consuming and can leave a bad impression that tests create a lot of effort for very little payback.

The return-on-investment of UI tests depends on the business circumstances. I'm guessing Boeing and Airbus have automated UI tests that sometimes break when programmers change things which causes rework. However, the pain of fixing the UI tests and keeping it sync'd with the UI code is worth it for avionics software.

Visual diffing in UI tests is something that actually can be solved with more code, see https://github.com/americanexpress/jest-image-snapshot, https://github.com/gemini-testing/gemini, or https://github.com/BBC-News/wraith.

> you should write tests for things you want to make sure they will not break

You should also write tests for things that are already broken before you work on the fix so that you can be sure it's actually fixed. Basically the red/green/refactor cycle[1] from TDD.

[1] http://www.brendanconnolly.net/test-driven-testing-red-green...