Hacker News Reader

Comment by BrianHV | Hacker News Reader

I have to admit I've always been puzzled by the opposition to TDD. I don't see it as a novice thing, or a dogmatic thing. Yet many in this community see it as such, so I'm wondering what they're seeing that I'm not.

To me, the fundamental benefit of TDD is that it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect. I'm not aware of any other technique that does that. And I find that to be compelling enough to use the technique routinely.

It seems to me the article author's basic point is that TDD alone isn't sufficient to prove all the requirements are met. I agree with that, but I also don't think that's sufficient to dismiss the practice.

View on HN

To me, the fundamental benefit of TDD is that it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect.

I think this is the sort of statement that makes some of us foam at the mouth and gesticulate wildly at well-intentioned TDD proponents. TDD doesn't prove anything of the sort. It proves that the code passed your tests before you made the change and passed them after - which is a very, very different thing from proving the change you made did what you expect.

I'm not aware of any other technique that does that.

Because there isn't one, in the non-trivial case but there are many that try to help - from static typing to design-by-contract and of course a zillion other testing methodologies. In fact, chances are very good that what you do is not strictly unit testing. And that's fine and good. TDD didn't invent testing.

http://blogs.msdn.com/cashto/archive/2009/03/31/it-s-ok-not-...

Another rather irritating thing is the insistence that TDD is a design methodology. If you haven't yet, read the informative case of an XP adherent trying to write a sudoku solver with TDD. Here's a blog post that links to all the relevant articles - the XP'ers attempt, Peter Norvig's solution and a reference to Coders at Work where Norvig talks about it a bit.

http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-s...

Long story short, if you don't know how to solve a problem and reason about a problem TDD won't help you. Neither will Design Patterns, UML or any particular fashionable technique, of course.

I think perhaps in my quest for concision I failed to communicate. I certainly didn't mean to imply that TDD could provide certainty for the entire system.

But I have to differ with you in one key point. TDD specifically proves that the code does not pass your test before you made the change. The red bar is, in my mind, the most important part of the practice.

Given that you have a failing test, you make a change, and then you have a passing test, what more do you feel is required for proof that the change did what you expect? It's possible that "proof" is too strong a word, but I find that process gives me sufficient confidence.

(Side note: I'm honestly trying to figure out the best way to write code. I don't think TDD is the silver bullet, but I think it's a useful tool in the mechanical part of development.)

We're using 'pass' in slightly different senses, what I meant is (and this applies to any test methodology) - in the strictest sense, getting the expected outputs from a test-suite, making a change and getting the expected outputs from a test-suite proves that you are getting the expected outputs from your test-suite. It may do a lot of other things but typically proving correctness is not one of them.

Side note: I'm honestly trying to figure out the best way to write code.

I think that's key and merits more than a side note. One of the reasons just about any methodology can boast of success stories is that given a concerted, focused and rational effort to improve quality, quality tends to improve, regardless of methodology. Such efforts are quite hard to get right, especially for organizations but when they succeed, there's little empirical evidence the primary cause is the methodology.

So if TDD helps frame your own efforts to be a better programmer, by all means, TDD away. The TDD griping comes from the often overblown claims of TDD proponents that TDD improves quality. 'Honestly trying to figure out the best way to write code' is a more valuable starting point than TDD.

I think the important realization in here is: TDD is a /tool/. If we fix the *DD == tool, we can reason that a good programmer has a toolbox in his head, containing a number of such tools, that is, a number of development and driving strategies which can be selected according to the problem at hand.

This is at least what I am working on right now. I have a solid grip on TDD, and I begin to understand where TDD works and where it does not (e.g. simple datastructures are nice, and things like katamorphisms can be well-tested pointwise. Basically the "middle tier" and "inner plumbing" of an application, but complicated algorithms, IO and such are problematic.

On the other hand, I am currently investigating the refinement calculus in order to derive programs from specifications. It is true that RDD (for consistencies sake: refinement driven development) is really, really hard when you try it on big problems and IO is again an evil problem lurking in the dark, but on the other hand, refining invariants has given me some really elegant and clean recursive solutions to some really tricky problems. I have to admit, it took around an afternoon to understand the result from refining, but I was able to throw just about any input I could think of at the problem and it just worked.

Third, there is probably some sort of KDD, knowledge driven development. Knowledge driven development occurs if you see a problem and just know the solution. In that case, you can just write it down, fix a few bugs and be done (or receive really obscure wrong outputs :)). Compare Norvigs sudoku solver for some pure knowledge driven development.

I think this makes sense and at least fits my current observations and explorations in the land of improving ones code-writing.

> I think this is the sort of statement that makes some of us foam at the mouth and gesticulate wildly at well-intentioned TDD proponents. TDD doesn't prove anything of the sort. It proves that the code passed your tests before you made the change and passed them after - which is a very, very different thing from proving the change you made did what you expect.

Would it have helped if the OP had said "made me more confident" instead of "proves"?

It would not have helped at all since it completely changes the meaning. 'proves' is a statement about the code, if true then we can talk about the influence of TDD on the code. 'made me more confident' is a statement about the author's state of mind - it lets us talk about the effect of TDD on the author's state of mind, which is a lot less interesting. Re-reading my code carefully or telling myself I'm awesome might make me more confident in my code changes but you rarely hear much about Careful Inspection Driven or Self-affirmation Driven Design.

I very much agree with this. I for myself think writing tests first or rather during development for well testable units as sweeping a number of small pebbles out of my way, so tripping due to trivial typos is less likely. Certainly, there might still be big rocks to encounter and I might still fall due to a large crack in the ground, but I certainly won't fall over due to small slippery pebbles.

In other words, I think TDD can be an amazing help at _implementing_ a solution after it was designed and planned and I think the ability to change code while the tests lower the probability of existing assumptions breaking unnoticed allows one to improve the existing design of a solution easily and significantly, while I also agree that designing and solving with tests only is awkward.

In case you want better examples where test driven development will fall flat on its face, take a look at the problem sets on http://uva.onlinejudge.org/ . You have 1-6 testcases in each problem, so there are enough test cases? :)

To me, the fundamental benefit of TDD is that it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect.

TDD does not prove your change did what you expect, at all. In that you are correct. However, TDD does provide a way to alert you if your change caused a side-effect you were not expecting it to. ie, passing tests does not indicate success, but failing tests alerts you to unexpected failures in edge cases you may not have considered otherwise.

That doesn't invalidate what you said at all, and in fact I agree with it, and think too many people probably equate passing tests with successful feature implementation, which is not ensured. But the fact that it does not serve the purpose you state, does not mean it does not serve any purpose at all.

I think there is some value in it, I just don't know that it is what many of the proponents try to push it as.

but failing tests alerts you to unexpected failures in edge cases: That should be "alerts you to some unexpected failures". That is, your new code could cause a side effect that is not caught by the test.

> To me, the fundamental benefit of TDD is that it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect.

That's called unit testing, and it's gets you to the same place. It's just TTD says when you must write the unit tests: before the code.

As an experienced engineer, I love unit tests for the freedom it give me later when changing things, but TDD just feels all wrong during the early phases. It prevents me from seeing the larger picture and adds "inertia" to internal APIs that maybe later turn out to be inelegant. But that's just me, others are helped greatly by TDD in their coding.

So my opposition to TDD is strictly personal. It's just not for me, but feel free to do it all you like.

How do tests written after you've changed the code prove the state of the system prior to the change? Or put another way, what technique do you use to ensure the test is exercising what you think it is?

I understand what you mean about inertia. If I'm not sure what I want an API to look like, I'll sometimes mess around with it without tests, then discard the work and rewrite with tests. I suppose post-hoc testing could be equivalent, but I find I usually write the code better the second time.

I think I pointed that out in my essay when I discussed how one limitation with TDD is that tests must fail, then change the code to pass the test. This doesn't let you add tests which are expected to pass but which are used to assess the validity of the code.

One way is make the tests fail, such as by changing the comparison from the expected value to something else, or by forcing an exception at the end of the code. Once the test fails then you know the test is being executed against the expected code, so then change it to make it pass. This is done without modifying the code. It's similar to what you would have to do if you depend on a third-party package which might be buggy. Code coverage is another way to check if the test is actually exercising the code paths you think it should. And yes, in some cases I will insert failures into the code itself, to convince myself that the test is also working, but that's only one of several possibilities.

While you make only a single iteration when developing APIs, I find that I many more than that to get the API to feel right, and that API development needs feedback based on implementation. I'll end up with a bunch of semi-tested but reasonable code, which I then test through unit tests, guided by my experience and assisted through code coverage (either manual or automated). It is much easier to make this code work right than it would be to start from scratch.

I'm also not sure that I would say "post hoc", since it really is a mixture of different types and quality of tests all throughout the development process.

Thanks for this. I'm going to start paying attention to times when I want to change the API but feel inhibited by the tests. I haven't noticed it happening in the past, but it's possible I dismissed the idea of changing the API quickly enough that it just flew past me.

When the code needs additional functionality, then yes, I'd say that TDD indicates you should create a failing test case.

But if you are adding additional test cases to prove that edge cases work correctly (and no code changes need to happen), then I don't see why you'd need to have a failing test case, and I don't know how you could argue for it.

"I don't see why you'd need to have a failing test case". It's going to depend on the case, but I've had enough times in Python where I've written (via copy & paste) a new test case and forgotten to change the name, so the new test replaces the old, that I slightly wary of trusting that new test code which seems to pass, actually did run.

On the other hand, if I have two parallel lists, one with input parameters and the other with expected results, and I add a new test case to those lists, then I'm not going to worry so much about the new test passing by accident because the likelihood of making an silent error there is quite small.

Under TDD you write tests not after you've changed code, but before you have written any code at all. Under TDD, all tests initially fail, as there is no code yet to allow the test case to succeed. You then write the code necessary to allow the tests to succeed. This is why tests in TDD are commonly referred to as a 'spec.' They serve as the actual specification for what to write next.

My take on it is that this is useful when implementing a well-known spec, like a communications protocol. It is less useful in an iterative design where the assumptions of the spec may not hold true at all later.

* I love unit tests for the freedom it give me later*

I'm curious, what freedom is that? I'm not sure I've heard anyone else say that about unit-testing.

If you have through test coverage, you can cut out a sloppy-but-functional prototype and replace it with a cleaner (or carefully optimized, etc.) version, and be reasonably confident that your code will still work. It gives you more freedom to replace parts of a project, rather than getting stuck maintaining horrible legacy code.

Of course, there are a few big ifs there. Writing very thorough test coverage is not a trivial undertaking, and may not be the best use of your time. Also, this says nothing whatsover about when you write the tests.

This matches my experience (legacy code, ugh), and I watched a Google tech talk in which the main SQLite author, Dr. D. Richard Hipp, spoke at length about how SQLite's extensive automated tests (http://sqlite.org/testing.html) have given them room to change the implementation of major internal systems several times.

Writing tests before the code can have advantages. Specifically, it forces you to look at what the code's external interface is, rather than how it will be implemented. This is important, because the simpler the interface is, the less cognitive load it puts on someone using it, even if that person is yourself.

But I agree that being dogmatic about requiring all code to have tests written beforehand is silly. Being dogmatic about most things is silly.

I'm not dismissing the practice, nor am I opposed to TDD. But your experience and mine are very different if you haven't encountered TDD zealots. In fact, as much as I hate to say it, I think zealotry is the hallmark of the TDD community, such as it is.

zealotry is the hallmark of the TDD community

Yes.

it proves that the code is in the state I think it's in before I make a change, and it proves that the change I made did what I expect

This advantage is less useful the better you know the system and the better you are as a programmer. One experienced programmer I know (30+years programming) said "When I know what to test I don't need the tests." When I started programming it seemed like I spent most of my time getting the code to do what I want, but that’s morphing so I spend more time trying to understand what I need the code to do. IMO, TDD is like using a good debugger, they can be awesome tools when you have a vague idea of what's going on, but once you really understand the code base you they slowly become less useful.

PS: The guy I am talking about was one of those "old masters" you occasionally hear about. One of my favorite stories about him was: "He once spent a week cleaning up someone’s code and sent it off to testing with some misgivings, which it passed and then failed in production. Apparently the testers had learned to ignore his code as it was normally a waste of their time to look at it, and after a few years this ended up biting them is the ass." So clearly testing is useful, but you need to balance cost and benefit.

"I'm not aware of any other technique that does that."

Well, as pointed out by others, TDD doesn't actually prove that.

There are however methods that do, which fall under the general heading of formal methods or program proof. Read "A Discipline of Programming" by Dijkstra for a reasonable introduction. The techniques work, but are laborious. Modern theorem provers can help, but it still involves considerable effort. Which implies that such methods likely won't provide a return on investment unless the cost of errors in the program is extremely high (avionics, medical control).

I prefer to think of TDD as similar to working a problem backwards and forwards in elementary algebra. You may still have gotten the answer wrong, but you'll have to have made a mistake twice, so it helps your odds quite a bit.

Though I don't have any opposition to TDD as a technique, I have an issue when some developers believe that just because the code has tests it means that it works and if the code doesn't have tests it doesn't work.

Reply on news.ycombinator.com