It's a fantastic comparison of the Detroit school of testing (what you and I believe) vs the London school, which is simply a different mental model about code and testing and mocks especially.
Framing the differences this way make it clearer why some people might disagree with us, and it's not because they're stupid or wrong- they just have a different school of thought on the matter. This doesn't excuse, however, the people who are doing a little bit of both and they don't know why, and they're getting the benefits of neither side of that coin.
For me, the biggest benefit is that I want to be able to refactor the internals of my code and have the tests (which map to business requirements, not implementation details) still pass.
It's also because a lot of the code I work with is inherently concurrent, and in my estimation, you're getting a lot more test coverage with the detroit model because a lot of your threading cases are simply covered if you use real objects and test it black-box style.
Because I like to test everything, not just little parts I isolate?
But if you read the article, you can see that this isn't the only school of thought.
Also, in dynamic languages, you might even have tests that don’t break when you change (or sometimes even delete) the dependency because the mock was correct. That leads to a false sense of confidence.
Having multiple tests fail for the same reason is a waste of effort, either now or in the future (when new features are added or the constraints otherwise change).
Sometimes you have to depend on transitive properties to keep your test sizes down. If a function received the data, and the function has been tested thoroughly, why test it again while testing the callers?
Of course I try to make those fakes at as low a level as possible, but there's still a substantial part of my code that can't be unit tested, and needs test programs in the actual firmware to check it.
Say I'm writing tests for lets say ShoppingCart.
ShoppingCart depends on an implementation of ICartDiscountCalculator (CartDiscountCalculator)
CartDiscountCalculator depends on a number of implementations of ICartDiscountStrategy (ChristmasCartDiscountStrategy, VIPCustomerDiscountStrategy, BulkPurchaseDiscountStrategy, etc)
Each of those, in turn has their own dependencies, perhaps webservice calls, db calls, accessing caches, etc.
The way I would test ShoppingCart would be to mock out ICartDiscountCalculator, and provide known responses.
Are you suggesting instead that you build up the dependency graph - i.e the actual implementation of the discount calculator, the strategies, etc - and only mock when it goes to external code (eg the DB calls)?
This would seem to make testing very very difficult - since each test needs to know implementation details of everything it depends on.
[edit] In other words, you don't test that "if discount is 10%, adding item to shopping cart increases cart total by 0.9×<price>"; you do test that "if user X adds item Y to shopping cart, and X has a 10% discount code, then shopping cart price increase by 0.9×<price>". This does seem like a headache, but it has benefits - in that it tests actual product scenarios, and you can now easily add variations like "if discount code only applies for books, and item I added isn't a book, then shopping cart total increases by <price>".
One of the problems I'm dealing with at the moment is exactly this problem - someone's gone and written a whole ton of tests which involve spinning up pretty much everything short of the DB, and now writing a small test that just tests the behaviour of one class is exceedingly difficult.
To take the shopping cart example I gave, if I write it the way that I understand you're suggesting, I would have to consider the behavior of all of the ICartDiscountStrategy implementations.
For instance, I've written tests for CartDiscountCalculator and ShoppingCart that have 10 different items in the cart from Acme Corp Manufacturing.
Some time later, I implement a new AcmeCorpDiscountStrategy which results in a different discount being returned. Now my tests for CartDiscountCalculator and ShoppingCart need to be modified.
I also need to potentially modify everything else up the dependency graph from CartDiscountCalculator - so my internal API that is used by customer service, an API used by mobiles, and the front end website.
Using Mocks/Fakes to isolate the ShoppingCart, it doesn't matter what the strategies are, we only care about how it interacts with ICartDiscountCalculator.
That interaction might change, maybe ShoppingCart uses the CartDiscountCalculator in a different way yeah, that'll break ShoppingCart Unit Tests - but it's that method only, and you're already in there modifying ShoppingCart anyway.
No, you would need to consider only the business requirements.
> Some time later, I implement a new AcmeCorpDiscountStrategy which results in a different discount being returned.
Why did you implement that? Is it a business requirement change, that "existing customers should get this different discount"? If yes, then your tests break - but for a good reason. If it's not about existing customers, but new sort of customers, then your tests should be unaffected and you just need to write new tests that target these new sorts of customers and validate that the new discount strategy is applied.
But, I'd suggest if you can't test a single class in isolation easily (and you might need fakes/mocks) then the class probably needs redesigning.
For your use case, if the shopping cart is simple enough, I'll just choose a simple or frequently used discount calculator and make some unit tests to test the shopping cart functionalities. Then for the rest of discount calculator cases there will be separated unit tests.
The problem with testing almost everything in isolation makes refactoring some small-inner components costly, since every change may break some of unit tests.
The question is why you want to test that one class? We do have some modules in our (elixir) system that are 'thoroughly' unit tested. These are modules that are algorithmically complicated so testing them in isolation is useful. Most of the code is tested in logical chunks though.
In unit specs you should use mocks/fakes for external behaviour. Then you can simply verify that it's calling the dependency correctly.
Isolation specs are useful because when integration tests fail, they usually give little feedback as to what exactly went wrong in the system.
Also integration specs are expensive and slow to run since they execute everything.
Finally I think mocks/fakes are useful because if you find yourself mocking 20 objects to get a single unit test working, then it's a sign that you need to redesign the code.
I would refer to these tests as "integration tests" although the language is poor here. Test a load of module together. This is beneficial in a number of ways: It tests whether modules are integrated together correctly - you can check that module A works and calls your fake of module B correctly, but you don't have a good way to check that your fake of module B matches module B behaviour properly (in practice it's usually when there are multiple modules involved that the problems occur). Unit tests only test stuff in isolation. It also means that you can refactor all that code, and as long as you don't change the top interface the tests will remain valid. If you move some responsibility around and change interfaces this can result in a lot of time consuming unit test changes, and that usually means that people don't actually structurally refactor as much as they should do.
Integration tests do not replace unit tests, they should just be taking a lot of the load. This at least is my interpretation of the well known "Write tests. Not too many. Mostly integration" tweet (https://twitter.com/rauchg/status/807626710350839808).
But that is an implementation detail. Only the BulkPurchaseDiscountStrategy knows whether it's 9 or 10 that's the trigger. Maybe it varies based on time of day or for certain customers.
> I would refer to these tests as "integration tests" although the language is poor here.
Agreed, it's a discussion I'm having at my job now, I havn't found particularly great wording. "Functional" testing still doesn't quite work right.
> Integration tests do not replace unit tests, they should just be taking a lot of the load.
Agreed on the first part. Integration tests are valuable, but I think they're more fragile.
Writing them is, IMO/IME, slower and more difficult.
As for the linked tweet, it's the "mostly integration" part that I disagree with.
Language is poor, but I think there is consensus on what "implementation detail" means, and I don't think that's it. Whether it's 9 or 10 is customer facing behaviour. It's behaviour that might come from your marketing team or product owner or whatever, but it is outward facing. It is something that someone could write in a word document and call part of your specification.
The implementation detail would be whether that logic lives in BulkPurchaseDiscountStrategy or as one case in a big ShoppingCart class. A user pressing buttons on your UI can see the result of the price break, but they can't see the difference in code organisation.
You can say "price breaks should occur at 9 items" without ever saying "BulkPurchaseDiscountStrategy". In one ideal, all tests would test user facing requirements and not know anything about your code. You could refactor your code to your heart's content and know you the result was still what you wanted. There are downsides to that though because those end to end or system tests are fragile to write and expensive to run. It's much easier to cover all possible code paths with more focused testing. Hence in this case you could test the top interface of ShoppingCart, with all it's assembled dependencies. Not the same as pushing button in a Web Browser, but a lot more convenient to write and run.
Point is that your implementation of the discounts is expected to change. I should not have to change my shopping cart tests at all just because the discount requirement changed - it makes the costs of those tests too high.
Of course this is domain specific. I expect discounts to change all the time. Likewise I expect taxes to change often. In both cases they need to be mocked/faked just to keep my test code sane. there are many other domain specific things that won't change. I'm always going to store the shipping mass in KG, and for some customers convert it to lbs. It is reasonable to hardcode the weight in lbs as that won't change. (future me who has to ship to mars will disagree, but I don't know what the requirements are)
* Change the (or shared test code) that has the price break specification from 10 to 11 * Run it, see the failures * Update the source code appropriately
The updated source code is the same in both our cases. In my case the changed test code is testing the ShoppingCart interface and in yours it's a unit test of something lower down. There's not extra cost, it's just in a different place.
Or are you suggesting that your tests don't know about 10 vs 11 at all? There's a reasonable architecture where such data comes from outside your system, and the spec for your system is 'price breaks should reflect this data input' and then you mock that data input and test it, but that's a different thing.
It might affect multiple tests, but all you should have to change is one or two places in some shared code pulled in by those tests. Too many tests have too much repeated code, and so become expensive to maintain, exactly like normal code. Refactor, think about shared functions etc. exactly as you would do with your production code.
Yes, integration tests are important too, but you'll want to minimize the number of scenarios you still need to test at that level.