Hacker News Reader

Comment by gleenn | Hacker News Reader

I feel like developers who don't write tests while writing their production code aren't doing a real job in 2024. Sure you might have a QA team or something, but that should be a vast minority of the tests being written. Test driven development and Agile and all that had some people upset and whiny 20 years ago, but if I see code without tests, I assume it's broken. The only way anyone knows anything works both now AND when the next 10, 100, 1000 commits roll in, your manually-tested edge cases have long been forgotten and I doubt you even got it right at all. Let's keep normalizing writing tests and thinking, up front, how to ensure our code works. That's the best time to think about all those pesky edge cases, and it's the only practical way to keep code working. Don't hoist this on to some poor chum because you are lazy or don't know how.

(Spelling)

View on HN

Most tests I encounter in the real world are terrible, fundamentally broken, and a waste of build times. I have almost never had a potential bug caught by a test written by others. I have been a software developer for 25 years across multiple languages and ecosystems.

Everyone jumped on the testing bandwagon but writing code that is testable is a learned skill that nobody bothers to learn. Instead we end up with overly-mocked tests that—in practice—test that “the code is written like it currently is” rather than that it actually behaves correctly.

These kinds of tests actually provide negative value. Besides taking up time during every PR and build, they fire off on any attempt to clean up or refactor. Every time you edit the code you have to edit the tests which entirely defeats the point.

The solution ends up being, unsurprisingly, lessons learned from the functional world: don’t access or manipulate external state, operate only on direct inputs and only manipulate your outputs. As a result code ends up being dumb, short, and obvious. But nobody learns to write code this way, so two hundred line spaghetti functions are the norm.

It sounds like you’re specifically talking about unit tests and people that suck at writing those.

I have seen functional and e2e tests prevent regressions far more times than I can count. The nice thing about those is that they force testing outcomes and don’t require “testable” code like unit tests do.

Don’t throw the baby out with the bathwater

It's really useful to ask the simple question "what tests could have caught the most costly bugs we've had?"

At one job, our code had a lot of math, and the worst bugs were when our data pipelines ran without crashing but gave the wrong numbers, sometimes due to weird stuff like "a bug in our vendor's code caused them to send us numbers denominated in pounds instead of dollars". This is pretty hard to catch with unit tests, but we ended up applying a layer of statistical checks that ran every hour or so and raised an alert if something was anomalous, and those alerts probably saved us more money than all other tests combined.

This reminds me of a fun thing I hacked up at a game shop. We were implementing some new feature involving a probability distribution and were somewhat stumped on how to make sure it worked right.

Eventually I setup something simple to exercise most of these functions, as well as most other balancing code, plot their results and send the results to the balancing team a week or so. For example, using D2 terms, it would run 1M item drops at different magic find levels and plot the resulting item level distribution, graph the implemented level -> hp/mana/... curves and such.

That little thing caught so many implementation issues at first and we'd regularly have balancing poke us because someone "optimized" some code in there.

Integration tests can (and often do) suffer from the same issues since being composed of poorly-testable pieces doesn’t make things any more testable.

I’ve come to the conclusion that mocks at all are evil when done for anything except external services not under your control. And then, what you should be making is a trivial fake implementation of the other end, not just checking that specific methods were called.

I mean, anything can be done poorly. But consider that the team that builds the vehicles you drive is not the same as the team that tested said vehicle for safety concerns.

That is, you can have a solid testing team for pretty much anything. You have to empower them, though. Largely, a lot of what you get to empower them with is ability to put constraints onto the engineering team.

This could be crappy constraints that are far sweeping on the product. There is no need that it has to be that, though. The only constraint you have to have is stability of product interface. Which.... yeah, our industry doesn't do that so well. (Note, not stability as in "doesn't crash." Stability as in, same inputs that worked last year work today.)

When I was doing safety critical software at the beginning of my career, we had a whole team in charge of building simulators replicating the API provided by the pieces of equipment we were integrating and their full behaviour. These simulators were themselves fully tested using both mock network traffic and recording of the real piece of equipment. This was used for integration testing and then repurposed for training.

Every testing environment I have seen since have looked like a joke but to be fair the testing budget of this place was bigger than what some companies spend on a whole product.

Yeah. I work for a database company, if we didn’t have copious amounts of integration, regression and stress tests we would absolutely be having a very bad time.

And of course we still have manual exploratory QA, it’s difficult to replace that safety net.

To be clear, tests when done well are incredible.

Very little in this industry is done well. Tests are one of those things.

> As a result code ends up being dumb, short, and obvious.

I did a presentation on that last year:

https://dconf.org/2023/#walterb

All this speak about whether we prefer to have no tests to bad tests is pretty pointless. It is a bit like asking whether you would prefer to be beheaded or hung. The correct answer is neither. If one wants some semblance of quality one needs BOTH tests and good quality tests. Let me sum up some attributes of good tests.

(1) They do not test at a level that is too low. If you test individual classes that do not have much logic in them, it probably is pointless. You are probably testing exciting facts like 'is the container type in my favorite language still capable of storing elements'. Mostly, it is best to test the interplay between a small number of classes on such a level that the attributes are such that the customer could recognize them as something that they value.

(2) Don't test at a level that is too high. Integration tests often take a lot of effort to set up, they are brittle, and they are slow. Have a small number of them but not too many. I.e., the concept of the testing pyramid.

In this thread one can read various assertions that sound like nonsense to me.

(1) Only write functional code: nice if you can get away with it but some code has the explicit purpose to manipulate things in the real worlds or is explicitly there to maintain some state. One might also watch https://www.youtube.com/watch?v=j71n33A0CkI&t=314s . The video is right. There actually is not much difference.

(2) Don't mock. Well, if your program is big enough to contain many classes and/or functions so that it would be too big according to criteria (1) and (2) above, it becomes impractical to test all of them together so you will need mocks. What can just be passed in as function arguments can be passed in as function arguments but what comes out as behavior may need to be mocked.

(3) Create test doubles instead of mocks. Use whatever is most convenient. Mocks record series of function calls. Test doubles maintain some state and one can see how that changes. Both can be good or bad depending on the situation. It seems pointless to have a preference separate from what you are trying to do

what would you recommend to read to be better at writing tests?

> a waste of build times

There's an assumption here that tests need to be run for each build. This rules out more laborious testing that could be exposing bugs not found by time-limited tests. For compiler testing, one can set up property-based tests that run for unlimited lengths of time. These can be set up to be always running in the background. Literally billions of tests can be constructed and run.

I understand this very much, but I have one big difference in experience: Once your coverage reaches about 60%, tests expose crashes simply by running the code. Even if there are no assertions.

I've tamed a few untested code bases by writing huge integration test-ish unit tests, and comparing the output with pre-recorded answers. After that, you start adding real unit tests for fresh code, knowing the existing stuff is quite safe.

I fully agree. For most of my career, I hated the idea of writing (unit) tests. They were almost always a waste of time.

At one point, I ended up on a team that had virtually no QA and only did automated end-to-end tests on a complex product. After working through a lot of tech debt on the testing (took a few months), it was solid and caught many defects introduced by junior/new developers (easily missed in code review).

I now have the fortune to be in a different group with a similar mindset. I’m motivated to write tests for my own code just to prove to myself that it works properly in odd error cases too. My team prefers testing instead of “run fast and break things.” (And with multiple decades of experience, I humbly can state I cause significantly fewer bugs than in the past.) This also makes me write code in an easily testable manner. Trying to test already written code is usually a lengthy and painful exercise in frustration.

It’s also too easy to write error handlers that are NEVER actually exercised once, even manually — and surprise, they may not even work! People who never write tests do this quite often.

Reply on news.ycombinator.com