Comment by paulmd | Hacker News Reader

Comment by paulmd | Hacker News Reader

Integration testing and unit testing aren't direct substitutes. You would never ever say "I have unit tests, I don't need to run the product before shipping it". Furthermore, automated integration testing is strictly less capable than a human doing integration testing.

Unit tests also have the advantage of being cheap to set up and cheap to run, so their limitations in whole-system testing are offset by the fact that you can have many, many more of them. You can easily blow through a couple thousand unit tests in a matter of 10 minutes or less, whereas even a modest battery of integration testing will be an overnight process at best. That means your OODA loop is a lot faster on unit testing.

(the ideal is of course structuring your application so you can deploy many tests in parallel, but that's not always possible.)

Fixing Selenium so WebDriver instances don't take 60+ seconds to launch would be a major help. Until then, integration tests that work at a controller level (minus the UI) are probably the best compromise between speed and integration. You can pass in a mocked request as input, and assert on the output. Runs much faster than a "true" integration test with a driven browser client, but hits the whole stack better than a unit test. They are also really easy to set up, since you can just have a tester or an integration test go through the motions and then dump request data to JSON, then feed it back as a test. The lack of state persisted between requests makes implementation a lot simpler.

"Unit tests also have the advantage of being cheap to set up and cheap to run".

They are not cheap to develop and maintain however. And given enough developers a certain percentage of them will completely distort the design of the code to enable unit testing making the code itself much more complex.

I'm having trouble thinking of how unit tests are "expensive to maintain"; you don't really need to do anything other than having them there.

If it's because you need to change them after a feature changes or breaks, isn't that the entire purpose of testing and not exclusive to unit tests?

Unit tests can be hard to maintain for several reasons. When you refactor your code (renaming methods, changing parameters, splitting methods up, etc), you will need to update those tests. Also some types of unit testing can be more brittle then others. For example if you use mocks to invocation counts, refactorings can be even more costly.

This comes up because there's a secret disagreement about the definition of the term "unit test". Some people see units tests as per method tests. If you have 20 methods in a class, all 20 methods get unit tested.

Others see units as the smallest bits of irreducible business or tech knowledge. So if you have 20 methods in a class, you test the 5 public ones, or the methods that compose a useful "unit". This is the definition used by the original test driven development book.

Many of the unit test advocates I've read use the second definition because testing private functionality incurs all the problems the article points out.

Using the second definition leads to the testing pyramid, which is a thing of beauty. Unit tests have a few mocks to the unit's partners. The unit tests test for the correctness of the unit. The integration tests are one level up in terms of code coverage in a single test, and they ensure that the mocks used in the unit tests are correct. The e2e tests are then used to test that the app starts correctly and is configured correctly. The obvious happy and sad path scenarios are also tested to ensure that dependencies are functioning correctly (in a complex application there can be hundred of "obvious" e2e tests).

In my career, every single instance I've seen about someone complaining about writing unit tests, they're either using the first definition I mentioned above, or their code is not composed correctly. I'm sure exceptions exist somewhere, but I've yet to see them.

Using integration tests like the article describes leads to a host of problems. For starters, it's extremely difficult to get complete code coverage without huge or redundant test suites. These large test suites are harder to read and take longer to write because more complex testing requires more complex mocking. Regression is also more likely during large refactors since you haven't clearly enumerated expected behavior.

Fun fact: in the glossary at the end of the original xp book, it describes unit tests as 'tests from the perspective of the programmer', and that's that - a woolly cop-out.

Thats a great point about about the 2 definitions. Many devs dont seem to get encapsulation at all and definition 1 might be related to that. If you expose everything from a module I guess both your calling code an your unit test are unclear of where the boundaries are

Those are all code smells.

Unit tests should only change if you also change public apis (I have one exception to this rule, which is very complex private methods which could easily be their own unit).

Then if you're changing the tests, you're also refactoring everything else.

>For example if you use mocks to invocation counts, refactorings can be even more costly.

These should only be costly if you're over-testing. You shouldn't be mocking and asserting calls on every call within a system under test, you should be checking the ones that matter.

Correctly written unit tests are a refactoring aid.

Reply on news.ycombinator.com