Today I Got Burnt by Isolated Tests
codelord.net
codelord.net
One word: Overmocking. Mocks should be the last thing you add to any test suite and they should only be used to mock out external services or slow systems. If you find yourself needing a lot of mocks for simple tests, then your design is bad and needs to be refactored.
Ryan Davis explains things best when he gets into minitest/mock (for Ruby): http://confreaks.net/videos/618-cascadiaruby2011-size-doesn-...
I've been bit by this before, multiple times. I've ripped hundreds of lines of mocking code out of tests in the code I routinely work on (not originally written by me) and it has given me much more faith in my tests.
It really depends what you're trying to achieve from your tests. I find a combination of all of:
1. Mockist tests that mock /all/ dependencies
2. Tests that stub out interactions with other layers (e.g. test the entire model)
3. Full integration tests that test the whole system with real dependencies (real db, real services)
works quite well.
The first set help you design and document your implementation through mock specifications.
The second allows you to assert the intended behaviour of the code without tight coupling to the implementation. You're less likely to break these when refactoring.
The third is the most likely to find real bugs in my experience. They're also generally more difficult to write and take longer to run, but valuable.
Mock-everything tests aren't good at finding defects but they're not without value.
In the end, there is no substitute for integration tests. We make it a point to write integration tests for atleast all happy paths.
Make sure you use integration tests (or a smoke test) as mocking away the entire environment will hide environmental and system errors when you actually run things.
- the size of the commit made it difficult to review it for potential errors, and - the feature, once committed, was not fully exercised, or as the article puts it, completely followed-through.
1. Unit tests - for regression testing and to enforce a clean design 2. Integration testing against real data - To check that there are no obvious bugs caused by bad interfaces or data 3. Smoke tests - To ensure that the core functionality works as expected
They also get updated in terms of frequency in that order. Smoke tests take the most effort to make, but they deliver the most value so long as they have supporting unit and integration tests which catch obvious errors.
This is exactly why I tend to be more of a "statist"l preferring black box tests. Yes, my unit tests are arguably a form of integration tests themselves. However, I can usually rest assured that my tests prove the behavior that they exercise.
The downside is that my tests run more slowly than isolated units under test.