Mutation Driven Testing: When TDD Just Isn’t Good Enough

Have you looked at Mull[1] or Dextool[2]?

[1] https://github.com/mull-project/mull

[2] https://github.com/joakim-brannstrom/dextool/tree/master/plu...

I'd be interested to see what a sufficiently strong QuickCheck specification of this problem would look like. I've used it a bit in the past, but not enough that I could reliably get it to produce all the interesting failure modes and know the expected result for each case.

Edit: To elaborate a bit, programming is naturally limited by the programmer's imagination, too. But what alternative is offered is a much more interesting topic than just disregarding something because it isn't a perfect solution.

[0] https://cs.gmu.edu/~offutt/softwaretest/

True, but in my experience this kind of testing tools do only complement testing. They should never be used to replace proper manually written tests as they are probability based and as long as the input domain is large enough it's quite viable to miss very obvious bugs not just in one rune but repeatedly.

Through if you are under time pressure it can be a good idea to, replace writing some relatively unimportant test with writing generative tests. Just only start doing so after you cover the most important parts with your manual tests.

It's kinda sad but for a lot of applications making the main features in their main use-case work right, and bringing out more main features, is more important then making all features always right but have less of them. In the end a imperfect but reasonable well working product (especially if all the small demo cases work) sells better then a perfect but very constrained product.

As an example. I was taught about object-calisthenics[1] in school. The lecturer treated a straw man version of it, and I thought about this like that for a while. What’s worse, when I later thought it was a good exercise and explained it to others I noticed I always have to reiterate multiple times that this set of rules is meant as an exercise of exploration, not a set of hard rules for whatever piece of code you are going to write next.

TDD is just so easy to turn into a straw man, and I think it has so many hardcore fans that makes it even worse to present the nuanced form of it.

[1] https://williamdurand.fr/2013/06/03/object-calisthenics/

If you know what result you want. It's easy to apply. Assert on the output.

If you don't know what you want, you can't apply it. Because you can't write the asserts.

So you can do exploratory implementation as long as you know the result you're looking for. But I'd also argue you want to have an idea about what you want before you start writing code. So I don't find that many places where it doesn't apply.

One of the few ones where it doesn't apply is tweaking the UI for looks.

Kent Beck recently went through and clarified TDD and he definitely doesn't advocate 100%. https://youtube.com/playlist?list=PLlmVY7qtgT_lkbrk9iZNizp97...

- Writing tests first.

- Have a write test => write code => loop.

But there are some people which are pedantic about how small the loop needs to be or insisting that TDD excludes doing any planing about aspects like how you likely will structure code on a larger picture and similar (i.e. software architecture).

At the same time there are tools/library/framework/language combinations which are testing in generally really hard and can play really bad with TDD. While I believe such tools should be avoided there are situations in which you can not do so and in which you furthermore due to e.g. time constraints are simile not able to do any proper testing including TDD. It's kinda a nightmare situations but it does happen.

EDIT: Yes I realized I responded to the wrong comment :(

While nobody proposes TDD "when it's not a good match", plenty of people overestimate the cases where they think it's a good match.

Plenty of TDD proponents believe you shouldn't write a single line of code without a test first. I've met them, and they believe it's a good match for the situation.

It's not helped by notorious failures like the infamous sudoku puzzle debacle -- a case where it was evidently not suitable, yet Jeffries went ahead and tried it anyway (and failed, predictably). The conclusion that TDD was not suitable for this kind of algorithmic exploratory development was somehow never reached...

This is the Nirvana Fallacy, "if it isn't perfect, it's useless". Fine, you can't predict all the bugs. But you can predict some, perhaps many. That is strictly more robust than not predicting at all.

> The author mentioned one aspect of TDD that has always bothered me: that I can’t “explore.” ... Rigid philosophies like TDD spike this methodology.

"Spike" is the word TDD weirdoes like me use to say we are exploring. TDD requires enough knowledge to specify the problem in a way that drives out code. When you don't have that knowledge, you explore first.

I've had codebases where I wrote and discarded untested code multiple times in order to understand what design I needed. Once I began to grok the problem, I backed out and then test-drove my way back in.

The idea here is to deliberately inject bugs to see if your tests catches them.

Tests are a (very) nice additional benefit, particularly because it has a simple way of ensuring coverage: only allowed to write production code if there is a failing test case, and only enough to make it pass.