Hacker News Reader

Comment by cjfd | Hacker News Reader

I learned TDD years ago and never looked back. It is true that there are some things where it is less suitable and therefore I would perhaps decide not to use it. E.g., user interfaces in case of changes that are mainly visual. Or code that tightly integrates with the system. E.g., file system manipulations. For all other code I think TDD is absolutely the best way to write anything more complicated than a throw-away script. Frankly, I am pretty much at the point where I consider anything less than TDD borderline unprofessional. I have come to expect code that was not TTD-ed to be buggy or more complicated than necessary or both.

View on HN

I take the opposite view. Code developed with TDD tends to be overly complicated and inefficient, and the test suites themselves are often bizarre labyrinthian hellscapes of mocks, facades, and workarounds. "They just did it wrong, then," you say? That statement ceases to be meaningful when it can applied routinely: it's the norm, and if it's the norm, it's not something "they just did it wrong" applies to.

So the way that TDD advocates point out that if your tests are ugly, its a sign your code is ugly. I think this also applies to languages and frameworks as well.

For example, I think of the experience doing TDD in Rails with all its ORM and callback interconectedness to TDD on functional code that is mostly exchanging values between systems. All the stub and mock crap goes away since values become the boundary of the system. Gary Bernhardt has an awesome talk making that exact point.

I'm now of the opinion that if TDD is a struggle, its a sign that your environment has serious underlying flaws.

>TDD on functional code that is mostly exchanging values between systems. All the stub and mock crap goes away since values become the boundary of the system.

That's generally why I think functional programming is taking off. It's much easier to reason about input-output of stateless functions. They're much easier to test and develop. They're way easier to understand for the reader, because the reader can understand what's happening in one place, instead of having to look through many different files, creating a complex model of all the inheritance relationships between classes.

"[I]f your tests are ugly, its a sign your code is ugly" is a bit tautological, and highly subjective besides.

I consider the 50-line function to be "less ugly" than the same function butchered for "testability" into a bog of poor abstractions and single-use smaller functions.

I agree some TDD advocates are way to big on loc as a metric for testability. Loc is pretty much irrelevant for both testability and readability.

For me at least, TDD has driven me to look for more functional solutions to problems. It's hard to come up with a more testable piece of code than a pure function that spits out the same output every time for a given input.

Unfortunately this approach tends to be less effective in languages that do not have good support for functional ideas like Java and C#.

Have you considered that maybe TDD is just a really good fit for your brain?

I've worked at a TDD-focused company and it always felt like I was coding through molasses. Some of us weren't as strict about TDD and I didn't notice a difference in our code quality.

I would be very curious to see the code base there. Some places are pretty bonkers about testing, in a way that seems almost religious to me. Excess mocking, extremely verbose tests, and a lot of cleanup to do in testing-land whenever you make significant changes in the production code. Maybe you were at one of those?

To me, doing TDD right is about having the minimum amount of testing needed to keep bug rates extremely low. It's also about keeping the test code as well factored as the production code, such that changing or refactoring the production code doesn't create disproportionate work in the test suite to bring it back in line.

An experiment I've tried repeatedly is to start a new project and shoot for zero bugs. I think I'm pretty smart, so that at first seems achievable without tests. But pretty quickly complexity increases that I at least have to manually test. And not long after that, manually testing everything gets tedious, so I have to start automating the things I am manually checking. If I keep pursuing zero bugs and feeling like I'm spending my time optimally, I keep ending back up at TDD. Maybe try that for yourself on a hobby project? You might find that the TDD-focused company was better at talking about TDD than doing it.

Everything you said is about good testing practice, not specifically TDD.

TDD can be nice for small modules when you have a good idea of where you want to go with your code. For large modules with a lot of unknowns it can be a pain, and in those cases I prefer testing after a first draft.

Perhaps you haven't built a module that was very large yet? Maybe try writing a large module with lots of unknowns. You might find that TDD gets in your way more than it helps in the early stages.

It's true that TDD and good testing practice are related, so I agree on that part of it. I also agree that sometimes when I don't know what I'm doing, it's worth writing prototype code and throwing it away or bringing it up to standards. Other times I'm fine starting with TDD as a way of exploring the design space.

As to whether or not I have enough experience with TDD to judge it, I'll leave that up to you. But I've been doing TDD ~20 years, and I wrote a lot of code before that. Let me know if that's sufficient.

I've found more problems with insufficient mocking as opposed to excess mocking. If you're not using mocks, you're likely using real dependencies. And that's where you get molasses testing. One small code change has cascading effects on tests, and small feature changes that take a few minutes to implement can result in a day or more of just fixing the affected tests.

Look up the Object Mother pattern. The idea is to have 'baked' objects which you can use repeatedly in tests, which fulfill a certain scenario. For example a hotel reservation system needs to be able to handle a single person up to a whole family (maybe with an allergy or special needs thrown in for good measure). So you set up a bunch of Guest objects using a GuestObjectMother.

When you make a change to Guest, you only need to change the GuestObjectMother and any tests that may directly be involved in the change to Guest. Tests that simply require a Guest to be involved in the test probably won't have to change, as they're simply retrieving the appropriate Guest object and then handing it off to something else.

It can go wrong both ways. But if changing real dependencies introduces cascading failures that have to be changed in a lot of tests, I'd suspect that there is expressive or structural duplication in the tests that could be eliminated.

> Frankly, I am pretty much at the point where I consider anything less than TDD borderline unprofessional.

Would you feel that way about the linux kernel for example?

Actually, I am not entirely sure how much of the testing there is automated. Presumably there are at least some automated test tools floating around here and there for the kernel. Major parts of the kernel could fall under the 'systems' exceptions where they are very close to the hardware. It is possible to do TDD to stuff that is close to hardware but it certainly is harder and it is more likely that some failures fall through the cracks when doing TDD. One thing that is different about the linux kernel, as far as I know, is how thorough the reviewing is that is going on for every patch. Also, there is the merge window followed by a long stabilization period. I would say that if you are not doing TDD you are going to need processes like that if you want to maintain stability. Also note that processes like thorough review by multiple people and a stabilization period four or five times longer than the merge window sound really expensive. Much more expensive than TDD. But also if one does TDD, still system/end-to-end tests are desirable. Not as much as without TDD but without it things would still break somewhat regularly.

With systems programming, performance matters, and all that extra indirection can be expensive.

Indirection can also make it harder to rigorously analyse all possible paths through a piece of code and make sure you’ve handled all required cases.

It's very possible to do the indirection at compile-time, set a flag only active during tests.

It’s possible, but typically you then pay a higher price on the other point I mentioned.

I used to work on a library that solved a certain type of mathematical problem, written mainly in C++. It was built separately for many target platforms in production: different operating systems, processor architectures, compilers. Each supported combination of these might require special provisions in the code due to compiler bugs, hardware limitations, etc. Then you’d have diagnostic code that was only included in the debug builds used during development. There are more than enough preprocessor shenanigans to go around in that kind of environment already, and another layer for test-related flags and checks isn’t going to break any records for most readable code.

in the languages I work with this is not a problem. Mox, for example, is what I use in Elixir, and even though "swapping out a module" is kind of a global change, it has facilities to track these mappings in runtime, even in concurrent testing, but the module is selected at compile-time, and only in test, so it is a "zero-cost abstraction" for running in prod.

I think this is just internalized pain of working with C++ (I suffered a lot with that 20 years ago). For systems-level stuff, I'm getting very excited about Zig, I imagine it will be basically "equally easy" to instrument these sorts of things at compile-time in Zig, as types are first-class values at compile-time. Zig is also super-composable; I have an example prime sieve that can run 60+ settings (combinations of single threaded, multithreaded, bool array, bitmap, hyperthread-awareness) by using a compile-time for loop across a series of instrumented settings; so long as I stay within the standard library the language itself already takes care of the polyfill for platform stuff, and I have access to those flags, if, say I wanted hyper-optimize for things like sse. I don't see any mocking libraries yet, but I doubt that instrumenting a mocking namespace in test will be any more difficult than it would be, in, say, elixir.

Tests != TDD. TDD encourages an architecture with lots of indirection so that it can achieve the necessary polymorphism for injecting dependencies in test.

I can assure you the linux kernel is not TDD'ed.

I think that's typically true, but that's because most TDD is done in languages where there's not a lot of cost for indirection. I expect we'd see a different outcome if something like a kernel were started with a TDD approach.

For example, we might see ways of expressing indirection for testing that then get removed when building the final product. Or we can look at the great strides made in virtualization since the Linux kernel was started. Something previously hard to test becomes much easier. And I'd expect more so if the testing had co-evolved.

Of course, there still might be hard-to-test areas. But given the massive ingenuity applied to the kernel development process over the years, I expect that they would have advanced what's possible in TDD if they had pursued it.

Why do you think that linux kernel developers have not changed their approach? TDD is not a new idea.

If indeed they have continued for decades to do it "the wrong way", imposing costs onto the project and society proportional to the number of preventable bugs creeping in, is this not a major scandal for the industry?

I suspect that the core team have reasons that they have not developed tooling to enable TDD of the Linux kernel. And that those reasons are more substantive than "meh".

TDD is not a new idea, but it's about 10 years newer than the Linux kernel. My experience is that projects very rarely make a switch like the one described by the Factorio team, so I assumed the Linux folks haven't. If you're saying they do work in a TDD fashion, I'm happy to take your word for it.

> TDD encourages an architecture with lots of indirection so that it can achieve the necessary polymorphism for injecting dependencies in test.

This is a language/library thing. Some language/libraries let you do TDD with very little indirection, a table of mocked dependencies as compile-time configuration, and all of the mocking events happen in code in your tests.

> One thing that is different about the linux kernel, as far as I know, is how thorough the reviewing is that is going on for every patch.

This is another underexposed feature of TDD: reviews. When I see a PR with solid test coverage, and those are green, it greatly speeds up the process. I just need to read through tests, see if they make sense, match the requirements and test those. And then glance over the added code quickly.

It (unfortunately, I might add) is not always possible to flat out reject a PR without solid test coverage, so it often is less work for me to add some test myself to a PR, than to manually poke around the changed code or to read through it in detail. This is not TDD, but testing-after-the-fact, and those test often are of much lower quality, but at least this is less work, for me.

Testing greatly speeds up the work for a reviewer.

If it started today, I certainly would. But I give them a pass because a) legacy code, b) legacy process, c) distributed process, and d) hardware is hard.

But consider this article on how it is tested: https://embeddedbits.org/how-is-the-linux-kernel-tested/

It mentions a lot of automated testing, which is good. And if you're doing automated testing anyhow, I think TDD is the most effective way to get there. But consider this:

"The kernel development process also proves that there is a big focus on testing since around 20% of the development process (2 weeks) is reserved for code integration and the other 80% (8 to 10 weeks) is focused on testing and bug fixes."

80% is a lot! And that also suggests significant lag between writing a bug and having to fix it, which means a lot more time figuring out what went wrong, and a lot more code written after the bug, meaning really fixing it might be expensive.

So I think it's reasonable to ask how much more effective kernel development would be if they could safely eliminate that 80% of time spent on cleaning up after the 20%.

I'm not a kernel developer, but I'm not sure if it's entirely accurate to call it an 80/20 split. Sure, Linus only merges new features during 20% of the release process, but as far as I can tell (again, not a kernel developer) most everyone else continues to work on new features outside of the merge window, and just submits them during that 20%, rather than spending that 80% only bugfixing.

Oh sure, I'd expect the labor numbers to be different. But it's still a pretty obvious constraint, and one that causes long feedback loops.

It also seems really ill suited for new systems and places where you're experimenting with new languages or frameworks. You don't know the shape well enough to be able to test the inputs and outputs before writing the thing, and learning via TDD seems highly suboptimal.

I think you have a fair point in regards to learning new things. YOu don't need tests to explore tools and languages. Tests are for development.

Thats just the thing with TDD: used properly (which is clearly tricky for many of us), it should help you think through what the inputs and outputs of the program you hope to write are. It does assume you know what you want to build though, even if you're unsure of how to build it.

I think of it as a design tool to be honest. When I'm not quite sure exactly how something should work, but I have a clear idea of what I want it to do, tests help me work through it in a way that produces extensible code.

It's not tools or languages you are exploring, you are exploring the problem you are trying to solve, and its possible solutions.

If you have a problem that is well understood, well documented and that has a straightforward solution, TDD is great because you know where you are going.

But a lot of the time, you have none of that. You don't know where you are going. You need to experiment, you need to iterate, you need to occasionally change directions completely. When you're doing that, TDD holds you back and gives you nothing.

In my experience a problem doesn’t need to be well understood, well documented, and straightforward to benefit from TDD.

Rarely are we building something with absolutely no idea what the final outcome should be, are we? There’s a general idea of, “this new feature should mostly work like this” description, correct?

If not, you’re doing code very different than the stuff I build.

And that might be the case! But I’m curious what kind of projects you’re working on where you have basically no idea what you’re aiming for.

Even when I don’t understand what’s wrong, I’ll generally have an idea of either: * what is undesired behavior to fix, or * new behavior to add

I can at least play around with the code a little to see how that impacts existing tests, or write some simple test that will demonstrate a really high level desired outcome. As I iterate the functionality, I might see the opportunity for another assertion or three, and then keep iterating on that until I’ve figured out what I truly want, both functionally and test-wise.

I agree with Karm on this. I'm not usually trying to figure out how to solve a problem at the first stage. I'm trying to understand what the app should do. I'm exploring the language or framework to understand how to pull that off.

Once I know, I can write a test based on just that information. The problems usually come well after that step as my implementation becomes more complex with the size of the app.

Yes, it's awful for spikes.

I have done a variant of TDD where the output is slightly uncertain where:

* I build a test with everything filled in except output of some kind.

* I write code that generates an output I want to eyeball.

* I run the test in "rewrite mode" where the output is generated by the program and saved with the test. I eyeball it to check if it's ok and then commit.

* In CI it checks the output against the fixed version.

There's plenty of scenarios where it doesnt work or is generally inadvisable but I find it to be a super effective technique where it does.

I think it's an approach that would work better for code that generates text, images, sounds, than common-o-garden TDD.

I don't entirely agree here, though.

If you are learning a new language, new paradigms or new frameworks, then, indeed, often testing is an extra burden that adds little at this point. But in this stage, I hope, one should be aware that the code you are writing is bad anyway. In a year, you'll probably be very ashamed of all the things you did there.

But in languages or paradigms that you are already familiar with, TDD helps a lot when learning new things. You can very easily learn about a new library by writing tests that use the lib. Or learn a new pattern by implementing them in tests.

For example, write some tests that wrap a Stripe library, then change the test to change some setting, call a method, or feed it some weird data, and assert some outcome rather than manually putting stuff in a cart, filling your address, CC details and then see that the setting you thought did X actually does Y. And repeat again.

Reply on news.ycombinator.com