Problems with TDD
dalkescientific.com
dalkescientific.com
Another issue I have with unit testing is that the cases which, for me at least, identify the most bugs pre-production are more integration-ish. Does the timeout setting for the HTTP API work as expected? Does this transaction really roll-back against an Oracle DB with the "thin" driver on JBoss? These are cases which are really hard to catch with traditional, use-the-UI testing, either manual or automated. I should read more about what others are doing for these semi-integration cases.
A couple of years ago, I committed a test case called TestHowDatabasesWork because of a team argument about how Oracle handled concurrency and locking. It got a good laugh, but it also documented a few behaviors specific to the problem at hand.
There is a good possibility that in documenting what you saw working you failed to notice some important details of its optimistic locking policy.
If you're ever again faced with such questions about Oracle I highly recommend picking up an appropriate book by Tom Kyte. For a developer in particular I would recommend http://www.amazon.com/exec/obidos/tg/detail/-/0072230657.
Testing the UI layer is quite difficult (at least in the absence of something like MVC, which too many developers can't handle and only want VB-like or WebForms-like coding). And in any paradigm I've ever heard of, real testing of the bottom layers of DB interaction is darned near impossible.
As the OP asserts, imposing a well-defined testing regimen is where most of the bang for the buck derives.
As for testing interactions with the database, one approach that I've seen work is to have data in your test cases with which you build up database objects that you then interact with. If your classes all have appropriate helpers and appropriate ways of inserting reasonable defaults on fields not explicitly set in the testing code, this makes testing the expected database interactions fairly straightforward.
Is it that because the code is written in an untestable way? A pretty standard mocking approach should be able to test this.
Until now I always assumed that the environment from which TDD emerged (presumably database-driven "Enterprise" apps) was different in a way that made it seem like a better idea than it does to me, but your comment is causing me to seriously rethink that assumption.
We were building (to be purposefully vague) an auction system where an auction could end at any point in time. Bids flowed in via a queue. It didn't really matter within reason which order the bids were processed, but we had to make sure that a bid which caused an auction to end was processed in isolation from other bids that depended on the auction still being open. Our code relied upon the DB's locking behaviors (SELECT FOR UPDATE) to ensure that the code processing a particular bid laid claim to the state (auction status, etc) used for its work. All fine and good except that we didn't quite understand how Oracle implemented locking at various isolation levels. So, some code which would have worked against, say, DBUnit, failed under Oracle. This wasn't a failure of the DB, but we wouldn't have caught this problem unless we ran an integration test.
Databases have always been one of my weaker points, and lately, ORMs have allowed me to be pretty lazy on that front. Thanks for walking me through that.
Subtle differences like these exist all across the various relational db world. Not to mention that they all do locking and concurrency differently, which is probably the biggest problem.
APIs should _not_ be from the perspective of "this is how I solve this problem" and should be from the perspective of "this solves my problem."
For example,
TempCalc tc = new TempCalc();
tc.f = 50;
System.out.println("50 degrees farenheit is " + tc.convertToCelcius() + " degress celcius");
This is an admittedly contrived example, but I see this principle in API design all the time. Everything should be from the perspective of the user, not the coder.
"I'd love to use this api if it looked like this."
Anyway... as far as that integration stuff goes, have you used Selenium? I'm not super up on Java testing stuff, as I'm a Rubyist, but I use it, and I know it has Java bindings of some kind...
On your API design comments:
My issue with TDD is that it's implicitly big-design-up-front. I used to think I understood a problem before I stated coding, but I don't anymore. So, anything I design in the beginning ends up being modified (if not scrapped entirely). There's no better way for me to understand a problem than by trying to solve it.
Also, for important enough APIs (public stuff, widely used in a codebase, or whatever), the design of the API is dependent on the problem being solved. Let's say you are building an API for a yet-to-be-developed photo storage SaaS. For any solution of significant sophistication, can you really think through all the issues that have API design implications before attempting to create the service itself? Is it better to stream back the bits of a photo found via a tag search or should a "ticket" in the form of a GUID be provided instead? I don't know, but I'll have a better idea once I spend some time solving the problem.
I guess it just comes down to personal taste, then. I feel that I can't understand how the coding should be laid out until after I've designed how it will be used.
TDD offers novice (or stuck) programmers two powerful nudges: 1) to break tasks down to trivial units, and 2) to think about what you're trying to accomplish before writing the code. Once you get the hang of doing this, TDD is just dogma to cling to.
To me, the fundamental benefit of TDD is that it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect. I'm not aware of any other technique that does that. And I find that to be compelling enough to use the technique routinely.
It seems to me the article author's basic point is that TDD alone isn't sufficient to prove all the requirements are met. I agree with that, but I also don't think that's sufficient to dismiss the practice.
I think this is the sort of statement that makes some of us foam at the mouth and gesticulate wildly at well-intentioned TDD proponents. TDD doesn't prove anything of the sort. It proves that the code passed your tests before you made the change and passed them after - which is a very, very different thing from proving the change you made did what you expect.
I'm not aware of any other technique that does that.
Because there isn't one, in the non-trivial case but there are many that try to help - from static typing to design-by-contract and of course a zillion other testing methodologies. In fact, chances are very good that what you do is not strictly unit testing. And that's fine and good. TDD didn't invent testing.
http://blogs.msdn.com/cashto/archive/2009/03/31/it-s-ok-not-...
Another rather irritating thing is the insistence that TDD is a design methodology. If you haven't yet, read the informative case of an XP adherent trying to write a sudoku solver with TDD. Here's a blog post that links to all the relevant articles - the XP'ers attempt, Peter Norvig's solution and a reference to Coders at Work where Norvig talks about it a bit.
http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-s...
Long story short, if you don't know how to solve a problem and reason about a problem TDD won't help you. Neither will Design Patterns, UML or any particular fashionable technique, of course.
In other words, I think TDD can be an amazing help at _implementing_ a solution after it was designed and planned and I think the ability to change code while the tests lower the probability of existing assumptions breaking unnoticed allows one to improve the existing design of a solution easily and significantly, while I also agree that designing and solving with tests only is awkward.
In case you want better examples where test driven development will fall flat on its face, take a look at the problem sets on http://uva.onlinejudge.org/ . You have 1-6 testcases in each problem, so there are enough test cases? :)
Would it have helped if the OP had said "made me more confident" instead of "proves"?
But I have to differ with you in one key point. TDD specifically proves that the code does not pass your test before you made the change. The red bar is, in my mind, the most important part of the practice.
Given that you have a failing test, you make a change, and then you have a passing test, what more do you feel is required for proof that the change did what you expect? It's possible that "proof" is too strong a word, but I find that process gives me sufficient confidence.
(Side note: I'm honestly trying to figure out the best way to write code. I don't think TDD is the silver bullet, but I think it's a useful tool in the mechanical part of development.)
Side note: I'm honestly trying to figure out the best way to write code.
I think that's key and merits more than a side note. One of the reasons just about any methodology can boast of success stories is that given a concerted, focused and rational effort to improve quality, quality tends to improve, regardless of methodology. Such efforts are quite hard to get right, especially for organizations but when they succeed, there's little empirical evidence the primary cause is the methodology.
So if TDD helps frame your own efforts to be a better programmer, by all means, TDD away. The TDD griping comes from the often overblown claims of TDD proponents that TDD improves quality. 'Honestly trying to figure out the best way to write code' is a more valuable starting point than TDD.
This is at least what I am working on right now. I have a solid grip on TDD, and I begin to understand where TDD works and where it does not (e.g. simple datastructures are nice, and things like katamorphisms can be well-tested pointwise. Basically the "middle tier" and "inner plumbing" of an application, but complicated algorithms, IO and such are problematic.
On the other hand, I am currently investigating the refinement calculus in order to derive programs from specifications. It is true that RDD (for consistencies sake: refinement driven development) is really, really hard when you try it on big problems and IO is again an evil problem lurking in the dark, but on the other hand, refining invariants has given me some really elegant and clean recursive solutions to some really tricky problems. I have to admit, it took around an afternoon to understand the result from refining, but I was able to throw just about any input I could think of at the problem and it just worked.
Third, there is probably some sort of KDD, knowledge driven development. Knowledge driven development occurs if you see a problem and just know the solution. In that case, you can just write it down, fix a few bugs and be done (or receive really obscure wrong outputs :)). Compare Norvigs sudoku solver for some pure knowledge driven development.
I think this makes sense and at least fits my current observations and explorations in the land of improving ones code-writing.
I think this is the sort of statement that makes some of us foam at the mouth and gesticulate wildly at well-intentioned TDD proponents. TDD doesn't prove anything of the sort. It proves that the code passed your tests before you made the change and passed them after - which is a very, very different thing from proving the change you made did what you expect.
TDD does not prove your change did what you expect, at all. In that you are correct. However, TDD does provide a way to alert you if your change caused a side-effect you were not expecting it to. ie, passing tests does not indicate success, but failing tests alerts you to unexpected failures in edge cases you may not have considered otherwise.
That doesn't invalidate what you said at all, and in fact I agree with it, and think too many people probably equate passing tests with successful feature implementation, which is not ensured. But the fact that it does not serve the purpose you state, does not mean it does not serve any purpose at all.
I think there is some value in it, I just don't know that it is what many of the proponents try to push it as.
That's called unit testing, and it's gets you to the same place. It's just TTD says when you must write the unit tests: before the code.
As an experienced engineer, I love unit tests for the freedom it give me later when changing things, but TDD just feels all wrong during the early phases. It prevents me from seeing the larger picture and adds "inertia" to internal APIs that maybe later turn out to be inelegant. But that's just me, others are helped greatly by TDD in their coding.
So my opposition to TDD is strictly personal. It's just not for me, but feel free to do it all you like.
I understand what you mean about inertia. If I'm not sure what I want an API to look like, I'll sometimes mess around with it without tests, then discard the work and rewrite with tests. I suppose post-hoc testing could be equivalent, but I find I usually write the code better the second time.
One way is make the tests fail, such as by changing the comparison from the expected value to something else, or by forcing an exception at the end of the code. Once the test fails then you know the test is being executed against the expected code, so then change it to make it pass. This is done without modifying the code. It's similar to what you would have to do if you depend on a third-party package which might be buggy. Code coverage is another way to check if the test is actually exercising the code paths you think it should. And yes, in some cases I will insert failures into the code itself, to convince myself that the test is also working, but that's only one of several possibilities.
While you make only a single iteration when developing APIs, I find that I many more than that to get the API to feel right, and that API development needs feedback based on implementation. I'll end up with a bunch of semi-tested but reasonable code, which I then test through unit tests, guided by my experience and assisted through code coverage (either manual or automated). It is much easier to make this code work right than it would be to start from scratch.
I'm also not sure that I would say "post hoc", since it really is a mixture of different types and quality of tests all throughout the development process.
But if you are adding additional test cases to prove that edge cases work correctly (and no code changes need to happen), then I don't see why you'd need to have a failing test case, and I don't know how you could argue for it.
On the other hand, if I have two parallel lists, one with input parameters and the other with expected results, and I add a new test case to those lists, then I'm not going to worry so much about the new test passing by accident because the likelihood of making an silent error there is quite small.
My take on it is that this is useful when implementing a well-known spec, like a communications protocol. It is less useful in an iterative design where the assumptions of the spec may not hold true at all later.
But I agree that being dogmatic about requiring all code to have tests written beforehand is silly. Being dogmatic about most things is silly.
I'm curious, what freedom is that? I'm not sure I've heard anyone else say that about unit-testing.
Of course, there are a few big ifs there. Writing very thorough test coverage is not a trivial undertaking, and may not be the best use of your time. Also, this says nothing whatsover about when you write the tests.
This matches my experience (legacy code, ugh), and I watched a Google tech talk in which the main SQLite author, Dr. D. Richard Hipp, spoke at length about how SQLite's extensive automated tests (http://sqlite.org/testing.html) have given them room to change the implementation of major internal systems several times.
Yes.
This advantage is less useful the better you know the system and the better you are as a programmer. One experienced programmer I know (30+years programming) said "When I know what to test I don't need the tests." When I started programming it seemed like I spent most of my time getting the code to do what I want, but that’s morphing so I spend more time trying to understand what I need the code to do. IMO, TDD is like using a good debugger, they can be awesome tools when you have a vague idea of what's going on, but once you really understand the code base you they slowly become less useful.
PS: The guy I am talking about was one of those "old masters" you occasionally hear about. One of my favorite stories about him was: "He once spent a week cleaning up someone’s code and sent it off to testing with some misgivings, which it passed and then failed in production. Apparently the testers had learned to ignore his code as it was normally a waste of their time to look at it, and after a few years this ended up biting them is the ass." So clearly testing is useful, but you need to balance cost and benefit.
Well, as pointed out by others, TDD doesn't actually prove that.
There are however methods that do, which fall under the general heading of formal methods or program proof. Read "A Discipline of Programming" by Dijkstra for a reasonable introduction. The techniques work, but are laborious. Modern theorem provers can help, but it still involves considerable effort. Which implies that such methods likely won't provide a return on investment unless the cost of errors in the program is extremely high (avionics, medical control).
I prefer to think of TDD as similar to working a problem backwards and forwards in elementary algebra. You may still have gotten the answer wrong, but you'll have to have made a mistake twice, so it helps your odds quite a bit.
More seriously, TDD doesn't really have nudge #2. I showed examples where TDD-based solutions weren't thought through well enough to be declared done, and I showed that in the prime factor case that the tests aren't sufficient to be good tests for the implemented algorithm. Specifically, tests should help others verify that the code works (which is not a goal of TDD), and the lack of tests with prime factors greater than 3 is a warning flag that should have been addressed during the development iterations. That the code happens to works (for sufficiently small numbers) is not the issue - the issue is that the tests aren't good enough to remove reasonable doubt.
However, for some people, I do think it does give them a nudge in the right direction. Over the years, I've been amazed at the quantity of code I've seen written by programmers who had only the vaguest notion of what the code they were writing was supposed to do. Any ritual, no matter how silly, that helps the novice (or lazy) coder to stop and think before coding is ok by me.
But, as I said, there's a long way from that to insisting on TDD as "the one true way".
But it's a long way from there to a categorical imperative.
The class of categorical imperatives, per Wikipedia:
A categorical imperative ... denotes an absolute, unconditional requirement that asserts its authority in all circumstances, both required and justified as an end in itself.
Whereas Kant's Categorical Imperative is an instance of that class, specific to morality:
Act only according to that maxim whereby you can at the same time will that it should become a universal law.
In the real world, everything is a shade of grey, and the same is true with software testing. Puritanical pursuit of theories and practices is silly. TDD is useful in some instances and not in others. A skilled and experienced programmer should be able to know when it's appropriate and when it's not.
Tests are no substitute for reasoning and TDD is just another fad.
Note that for this to work, you have to be writing tests based on the real use requirements. Simple synthetic unit tests will not help you here, and may lead to exactly the problem the author is complaining about.
One thing (really two things that work hand in hand) that I've found to augment my OOD skills so that I write better tests and better software is domain driven design and behavior driven testing. Using these techniques together has helped me to learn to ask better questions of my domain experts, create more decoupled and flexible software, as well as solve that age old how do I test the UI when it isn't already in an MVC framework (the correct answer is to treat it or portions of it as their own domains imo).
This is the kind of article that provokes thoughtful discussions by having a well thought out opinion and not just blabbering the first idea that comes to mind.
Thanks and keep it up!
1. If (all tests pass), Do one of the following:
a. Write a failing test
b. Write a passing test
c. Refactor
d. Start a 2 min discussion
Fix the code.
When you write your tests first, you will naturally never end up in that situation.
Of course, when you see that you have a hard time at writing unit tests because, well, the units are too big, then you can (and should!) start refactoring, but how do you refactor with confidence when you don't have tests to see whether the refactoring went well?
Chicken and egg I guess.
If you care about testing, use a quickcheck clone. Bonus: it's much easier to refactor tests when you're saying "it takes an X and a Y" instead of "it does this and then that". And if you discover a bug caused by a corner case of Y and add it to your Y simulator, then everything that takes a Y gets tested with that corner case.
just get things done, man!!
TDD is about design, the article is actually criticizes unit tests. TDD helps you achieve SOLID principles, which really helps in real world. Fibonaccis, primes etc. are only for demonstration purposes.
And 100% coverage is for sissies, only a TDD noob talks about it in this context since it's not sane.
Depends on your project and the cost of failure. I once worked on a system that would automatically instruct semi-truck drivers to move their vehicles hundreds of miles. I spend over a week building tests for transactionality/concurrency edge cases to make sure that two trucks could not be instructed to go to the same place in error. Worth it? Yup. Should you test all possible behaviors of a utility script that, if broken, can just be re-run? Probably not.
SOLID applies to OOD, and an essay I don't plan to write is why I consider OOD to be the wrong solution for most cases. In any case, my point is that TDD adds very little to the process of developing good software.
I mentioned several times that TDD provides weak tests and is insufficient to guide the development process. With examples. I did not criticize unit tests and I said that that's what I use in my own work.
The 100% coverage reference I gave comes from Kent Beck. I'm surprised that he's considered a TDD n00b.
You are more than welcome to reply in more detail.
Some people combines Scrum with their existing process, some set 4 months iterations while others insists on not more than a month. Some can't live with a Scrum Master who is a part time developer. By the book, they all wrong. Yet, they're successful Scrum implementers.
So here's my account of TDD, regardless of what Mr. Beck said in the book (I am actually an Alt.Net follower):
Unit tests and TDD, for me, are totally different things. TDD is a way of construction, while unit tests are for inspection. Unit tests' re-validation power is great, but less and less needed if you have a good design overall.
Hence,
- TDD should not care to emphasize good test cases. It shouldn't even emphasize a complete test suite against a method. If someone want to test every possible way a function can be called, there are even automated software for doing it. It's a QA thing, not design.
- Worst case scenarios should be left to your test framework. Same as above. Row attribute in MbUnit. Range, Random, Combinatorial etc. attribute in NUnit (I'm from .Net world).
- No approach should give someone confidence that his code works all the time. Even the best QA can't catch all. But "When the bar is green, you know where you stand." means you are safe on what you test. If you test only for the correct input, you're safe for it. You'll know that missing inputs are untested. (Just a couple of unit tests, though)
- There's no 100% coverage. Since 100% coverage might also mean a <div>'s positioning. Screw it and all the UI breaks loose. Critical paths should be covered as much as possible (some 80/20 rule applies here. %20 of tests cover %80 of critical path etc.).
- When you want to change your api, do so. Write new tests, then a new api. The tests you have written are not part of your code real estate. Either feel totally ok for throwing them away, or don't use them. You will break your mental model of the old api when writing the new one without tests (we all write unit tests in our heads when coding), so what's wrong with breaking the digital representation?
- SOLID is nice, don't overuse OOD. But, what's wrong with having OCP everywhere all the time? Need OAuth? Extend your account service. Want to also publish to twitter from your app's main feed? Extend feed service.
Unit tests can be written good or bad. It depends on one's ability, testing framework in use, even the mocking framework. But in my experience TDD only improves your code, makes it more modular, more open to extension, more open to change.
Worst case scenarios cannot be left to the test framework neither can good test cases. Leaving them out means the code is going to reflect an optimistic view of the requirement. A more pessimistic view may well entail extensive changes to the code, just like a more optimistic view may well be a lookup table if only two values are tested.
I never promoted the calling a function every possible way. In most cases that would take too long to be called a unit test. On the other hand, if there are only two possible ways then I would go ahead and do those as unit tests.
When I talked about "confidence" I referred to it as minimizing doubt. Other than a few cases of formalized testing or complete enumeration, I agree with you that you cannot be confident that the program works, if the interpretation of confident is "100% certainty." However, confidence has multiple shades of meaning, such as "I'm confident that I'll live to see 2010" when I know that that's not 100% for sure. In software the statement of confidence is more "I'm confident that the software is deployable with an acceptable risk and cost of failure." I don't think TDD leads to that level of confidence.
My position is like yours - 100% coverage is rarely worth the cost. I managed to get 99.8% coverage in a code base last year and failed to get 100% because I couldn't test two malloc failures without a day or two of additional work which clearly wasn't justifiable. My comment was the weaker statement that TDD does naturally lead to 100% coverage.
I wasn't talking about changing a developed API, my example was coming up with the API in the first place. If I develop the tests while I experiment with API and possible implementations then it's extra work with little benefit, and an obvious cost of time spent to develop those tests and modify them as the API changes.
What's wrong with OCP? As far as I can tell, what's wrong is the name. How is OCP different than "code reuse" and "reusable software libraries"?
I've just now done some shallow research on this. Meyer's original form promoted implementation inheritance, which is one of the forms of code reuse I've done. Martin's form of OCP promotes abstract interfaces, and I've grown prefer concrete interfaces. He specifically eschews public attributes and global variables, as being against OCP. I use them, but then again I also use Python, where these have different meaning than C++.
I'll also point out that Martin's form of OCP mentions "software entities" but really seems to mean "classes" based on the examples he gives. Template metaprogramming and Lisp macros seem to be two of many techniques that aren't mentioned in his discussions. For less esoteric topics, neither is Javascript code inserted into a web page during HTML template evaluation, nor dynamic code generation (eg, with LLVM), nor any place which takes an eval(). Those seem to be cases which can be quite useful and yet are counter to OCP.
As for your last paragraph, you've said that unit tests are totally different than TDD, but here you seem to say they are part of the same concept. My point is that good software development styles give the consequences you mention there. While TDD is one method people have used to learn good development technique, it isn't the only. Other development techniques must be learned, and these are not part of TDD as it's taught. My assertion is that TDD adds little to those techniques and as such isn't at best a weak concept.