Comment by spellboots | Hacker News Reader

Comment by spellboots | Hacker News Reader

It seems like there are a sea of absolute assertions with few facts backing them up - people seem to love either saying that TDD is better or TDD is worse, and it seems that the only data ever really offered on the subject is anecdotal opinion.

We can do better than this. This study[1] at Microsoft reveals that TDD results in a defect reduction rate of between 40% and 90%, at a cost of 15%-35% increased development time. So, what's the best approach? It now becomes a business question. If speed of delivery is crucial and quality less so, don't use TDD. If quality is more important that speed, use TDD.

And finally, everyone involved in building software owes it to themselves to take an hour to go and watch this excellent video by Greg Wilson entitled "What We Actually Know About Software Development, and Why We Believe It's True" [2], and then perhaps dive in to the list of references from it [3], because you will find yourself more actually informed than anecdote and opinion will ever get you.

[1] http://research.microsoft.com/en-us/groups/ese/nagappan_tdd.... [2] http://vimeo.com/9270320 [3] http://www.gdb.me/computing/citations-greg-wilson-cusec.html

As the other poster pointed out, this paper is measuring nothing, or at least it is not written in a way to let us know.

I find it unremarkable that a sudden focus on quality and testing reduced the bug rate. It does not follow that TDD is the cause, nor that TDD is the best of multiple options.

Its a well known effect in experimental design - measurement alters what you measure. Get depressed people to watch and talk about a dog video. I can predict that they will be less depressed. Not so much because dog videos are great at reducing depression, but because all of a sudden so much experimental interest is being directed at them. It would be almost churlish to not feel better, if you know what I mean. So, we don't design experiments that way - if your hypothesis is that dog videos cure depression, you need to do exactly the same protocol, but with cat videos, or I don't know, truck videos, or whatever, so both groups get the same attention, both are trying to reduce their depression and so on (I'm assuming unblinded here because obviously the TDD paper was unblinded, obviously we have better methods for the depression study than my suggestion here).

Kudos for bring empiricism into the discussion, but I do find the paper lacking.

I wrote above that I don't use TDD yet achieve low defect rates and good design. I think it is because, no matter how you do it, if you focus on those things you will more likely achieve them than if you don't. I hypothesize (don't be mad!) that TDD works for some simply because it brings the issue of quality to the forefront of their mind. Whereas, I'm already always thinking "is this line of code going to kill somebody"; I've tried TDD, and it always seemed like it added little to actively inhibited me. Your mileage will vary.

I've read the Microsoft paper, and I'm not clear on something:

The paper is comparing TDD vs non-TDD groups... but what are the non-TDD groups doing? Are they test last? Are they ad-hoc? Do they test at all?

Did I misread something?

Also, Greg Wilson's video is excellent.

Thank you for Greg Wilson's talk.

I've been trying to find empirical information about software development and he's frank about the lack of it. He noted work by Lutz Prechelt [http://www.inf.fu-berlin.de/w/SE/OurPublications], for one.

Reply on news.ycombinator.com