The maintainability of unit tests
blog.jayfields.com
blog.jayfields.com
If you are writing functional tests, the same thing applies. For a UI, you might have an API that reads like. "Go to the home page." "Create a User". "Login". "Go to the page I want". "validate this one behavior".
Since all tests are using an API, you just change the implementation of the API call. For example, if I change the number of fields required to login, I only have to change the login API call. For all the tests that used it, they don't care about how you login, because that wasn't the thing they were trying to test. It was just a pre-req.
Your refactor is not unguided, though, because you still have tests for the thing that consumes the unit that you are refactoring. That's how you know you didn't break anything -- the tests for the rest of your program stay the same during the refactor. (And you write new tests for the details of the code that you are changing, so that when you change something that depends on, you still have tests.)
A technique I find helpful is for classes to generally not call methods directly on objects that they old references to. Instead of calling $self->foo->bar, delegate $foo->bar to $self->bar. Then when you get rid of foo, either by refactoring, or lack of need in a subclass, you still have a bar method. Something else can provide it, or the class can provide it for itself. If you follow this rule throughout your application, classes will only be coupled to themselves; and this means that more tests survive refactoring. (Use of this technique is pretty tedious if you don't have a language feature to do the delegation automatically. I use Moose, and it makes delegation to has_a members trivial. Other languages are not so lucky.)
And I'm not spouting propaganda here, as I know this is a talking point, this is my personal experience. Take away this benefit and my desire to write the tests would be much less.
Before unit tests, I would do design one, build a bit, discover I need design two, build a bit more, discover I need design three, build a bit more, discover I need design four but now design three is really embedded too deeply into the system to know whether I can safely change things, and then the hacks start going in because otherwise the subtle bugs start popping up faster than I can catch or fix them. Unit tests don't eliminate the bugs popping up unexpectedly after a design change, but in my experience it brings them down far enough that the cost/benefit of the design change is positive again, which in the long term pays off when we actually end up with the rather well-tuned, powerful-yet-simple design thirteen. (Someday I hope to learn enough to get there sooner, like, design six or something.)
Am I just lucky, or are they doing something wrong?
On the front of what I do differently than some, I actually reject the "unit testing must test as small a unit as possible" dogma and the vast bulk of my unit tests are actually what some people would call integration tests, though, usually, rather low level integration tests; they don't take 20 minutes to run, either. I also avoid mock objects like the plague, aggressively preferring to either build the interface to be testable in the first place, or failing that, adding some hooks in the tested code itself, rather than trying to push it onto the code from the outside. Perhaps that's the difference... basically I tend to run my testing one level above the bottom layer on the real-as-possible code, rather than right on the bottom layer of the code with various fake bits attached. Despite what intuition (and dogma) might be telling you, this usually means I can change that bottom layer quite significantly with little to no test code changes.
(I don't follow this religiously, there is some bottom-layer code that gets tested on-the-spot, like raw validators or the output of a parser, but typically the code I test on the bottom layer never subsequently changes much because it is so simple that there's simply no reason for it to change.)
I'm not convinced that letting testing drive the design process is always a good idea (though the benefits of test-informed development are harder to argue), but having thorough tests is usually a huge win. Having thorough tests isn't free, particularly with good coverage, but it usually pays for itself in the long run.
> I also avoid mock objects like the plague, aggressively preferring to either build the interface to be testable in the first place, or failing that, adding some hooks in the tested code itself, rather than trying to push it onto the code from the outside.
Same here. Mock objects can be useful when trying to retrofit tests onto existing code, but that's just a concession to practicality. Reworking a huge, untested code base to make it testable can be a great way to introduce bugs unless done with extreme caution. (That's the main topic of Feathers's book, FWIW.)
Yeah, I gave full-on TDD the college try and did not like it. IMHO it might not be a bad crutch for those who are still new to code design issues, but it just involved too much closing my eyes to issues that I knew would be a problem in the future and willfully blundering into design traps I knew about in advance. Using it to learn about those issues is probably a good thing, but it must eventually be discarded.
I still do sometimes write the tests in advance when it makes sense (usually those same low-level things I referenced in my first post) but as a general principle I don't find it gives me the best APIs in the end.
On one side, TDD made sure that your code is designed to be testable at maximum level. That means: developers have no excuses to design their code to be testable (whether it uses DI, less coupling, or whatnot).
One thing about developers is that they're too self-confident that their code is not buggy. When bugs found, they just fixed it and didn't bother to analyze and learn from mistakes.
So what happened when they wrote untestable code and suddenly 2 weeks later there were bugs manifest due to whatever reasons? If the untestable code is easy to refactor, management probably won't mind a bit of hickup in their schedule. But if it takes a wee bit more time?
I think TDD is good when the developers are inexperienced. But at the same time, it's human nature to be lazy and made mistakes (or underestimate things).
I don't know the best answer whether TDD is good or not.
Testing upfront is just another tool, whether by unit tests, static typechecking, or a REPL. Often useful, but not a panacea.
There's some test setup stuff written in C and maybe C-level interface tests, but I think the functional tests are what lets him modify his optimizer, rather than some unit tests which are tightly bound to a specific implementation.
No, you've just found the workflow that works best for you. The problem is that it doesn't work best for everyone else.
No, you've just found the workflow that works best for you. The problem is that it doesn't work best for everyone else."
This. I read Jay's post as elucidating what worked for him (and no more). I didn't get a "preach the One True Way to the great unwashed" vibe from his post. Just as you have practices that work for you, he has practices that work for him. They are different from yours.
So? Why should it "boggle" your mind? It is not as if building software is a hard science. Different things work for different people and projects.
I read it as his reporting a consensus between some significant number of people.
"At speakerconf 2010 discussion repeatedly arose around the idea that unit tests hinder your ability to refactor and add new features.... Should I write Unit Tests? The answer at speakerconf was: Probably, but I'm interested in hearing other options."
It is the consensus that surprises me. No one person's opinion surprises me; you can find someone to believe anything.
How is "probably" a "consensus" ? It is more likely that there were people arguing "yes", some people "no" and others saying saying "it depends".
I would be very surprised if the folks at SpeakerConf (I've worked with many of them including Jay) arrived at a consensus that unit tests hinder refactoring and/or the ability to add new tests. But, yes, they are some very independent thinkers in that group and I would be very surprised if some of them don't think thoughts at variance with agile orthodoxy.
Jay says nothing about any consensus anywhere. Are you sure you aren't reading more than what he said into the post?
Well, "probably" would tend to imply the ones in favor of the thing labeled "probably" were winning. Or at least louder. If it was totally even, the word would be "maybe" or "nobody could agree". But:
"Jay says nothing about any consensus anywhere. Are you sure you aren't reading more than what he said into the post?"
In a word: No. But double and triple checking I still don't find it an impossible reading of the text, so I don't feel bad, and still won't even if it is clarified later that I am wrong. In the meantime a good discussion has broken out, so shrug.
Well I wasn't trying to make you feel bad, fwiw. Just wondering what in Jay's post caused you to say
"This discussion boggles my mind,".
I didn't see anything "boggle worthy" hence the question.
peace.
What would make be boggle is that a consensus would develop that unit testing was more trouble than it's worth most of the time; if no such consensus exists then I wouldn't be boggling.
I have to admit that I wrote unit-tests so that when other (lazy) developers change code that he/she didn't test, my unit-test will fail on his/her code. I like to view unit-tests (or integration, or whatnot) as a trap for these developers.
The thing is... this is the kind of people that I had to deal with: lazy to test their code completely, write premature optimize code, have a bad taste in designing classes, don't understand OOP (write code in procedural style), and think that their code is "better". It's hard to reason with someone who thinks that Singleton is great because it saves memory even if you told the person that it's negligible. Or lazy instantiation of a meaningless member variable is great because it shaves 40% of object construction when the actual number of creating object is meaningless. At the end of the day they would say "but it's better... here's some numbers".
I don't think I will change my mindset anytime soon. Especially when a couple days ago there were discussions on Reddit regarding "bad code", "bad programmers", and etc. I don't know how bad the situation in our industry as of right now because I'm seeing people writing more blogs/articles in favor of unit-tests (TDD or not) but from my personal experience and point-of-view, I saw a few good (and humble) developers out there and too many cocky ones.
PS: It's worth to mention that based on my personal experience, good developers are usually the ones that have a family, e.g.: Paul Buchheit, Joshua Bloch, etc. They seem to be wiser and more patience.
I'm not sure I understand this. How do you deal with things that write files out or communicate with other processes? That's where I've found mocks to be of the biggest advantage.
The problem with mocking that I've run into is that you're not really testing expected functionality. You're testing that code calls an API. This sounds good in theory until you need to change the API. Then, you have a whole bunch of tests to clean up.
It can be a pain to change the mocks and tests when the API changes, but what else can you do - not test this layer of your code?
I have the tested code write the files out, then I have the test code examine the files. I have the tested code communicate with the other processes, then examine the results, possibly querying the other processes to see whether they got what they should get and did what they should do. I make it so it's easy to put the files in a temporary location, easy to start up test processes for the test to communicate with, and then test the system at this higher level.
I am aware that this is counter to the unit testing dogma. I am aware this requires certain amounts of access to the code of all parts of the system, but I usually either have it or make it a priority to get it. So far, the predictions of doom if I do this have failed to manifest.
And I've learned that when someone predicts doom that fails to manifest, that's usually because the predictions are based on untested theories and not actual experience. This happens to an absolutely shocking degree in our industry; the amount of software design dogma that is basically just one guy with a PhD's opinion he stated in a paper once 20 years ago, followed by a huge game of telephone, can hardly be understated.
(Your homework exercise, if you don't believe this, is to find the published, peer-reviewed paper in which it was determined that unit tests work best if they only test the smallest possible units. Good luck. If that doesn't float your boat, pick your favorite bit of OO-design dogma, like the Law of Demeter, and trace back where it came from. Even good ideas generally don't have the sort of actual support you might think from how far the ideas tend to spread. My take-away from that is that not only is it OK to question, it is a necessity.)
Everywhere your test code tests something other than the actual code is a place where bugs can hide. Mock objects are basically a systematic way for providing those places for bugs to hide. I agree with silentbicycle that they have a small place, but IMHO systematic use of them is wrong.
I've come to the same sort of conclusions as the OP myself. To me, the most important thing is the code, run test, cycle. If it's too slow, I get impatient and easily distracted. That happens when the code base just grows and grows. I'm not saying don't test. But test the weird corner cases and normal cases that you need to get right.
It's a tradeoff between having complete tests and having fast tests for me.
1) I've found that potential bugs, and design problems, are often easier to spot at a lower level. Because of this, I've started using more integration-y tests to build out my initial implementation, and more unit-y tests to fill out additional details.
2) I've found the GOOD use of mock objects: when I have a class that wants to expose its data by calling a method on a narrow, well-defined interface, rather than by providing it through a getter, I'm happy using a mock version of that interface.
I wouldn't advocate it as The One True Testing Dogma, but I thought I'd share it in case you find it useful.
To put it another way, test those invariants that are least likely to change in the future. The "the e-mail sending component should properly construct the headers" test is testing an invariant that's unlikely to change. The "the EmailComposer class will call the EmailHeaderConstructor's constructHeader() function" test is testing an invariant that's much more likely to change.
The problem is that many unit testing dogmatists (and the associated books and conferences) encourage people to write lots of tiny little tests for every little piece of everything, and it's those tests that kill you when you're refactoring.
If you have a logical component that consists of 10 different classes that work together, your main set of tests should be at the component level, and those classes should be treated as implementation details that are only unit tested if they themselves are complicated enough that they likely have errors that aren't caught by the larger test suite. If you go crazy and unit test all the interactions between each class within that component, then when you later decide to re-architect parts of it to improve performance or simplify the code or fix bugs or in preparation for new functionality, you end up with a bunch of tests that you have to either fix or delete.
Tests like that end up discouraging refactoring due to the friction they exert, rather than encouraging refactoring the way less-fragile tests do.
Secondly I think we may see a world where test suites are written by the higher level engineers and the implementation is farmed out to the cheapest bidder. What if there comes a day where we don't even read the actual code any more? Just look at the test report for correctness and that it performs well? In that situation I can imagine the test suite becoming the best design document that software engineering has produced.