If someone finds it burdensome to review my 50-line AI-generated bug fix, adding 200 lines of tests only gives them more lines to review.
Unit testing imo is pre-LLM. It was mostly to keep the human honest because the human can't possibly read all the code to make sure nothing broke.
Are you maintaining a list of all invariants you want QA'd by the LLM after each change? How many tokens does this process take?
Claude Code (Opus 5.5 xhigh), when it makes changes, tests the relevant endpoints for behavior based on what it touched. You could also just explicitly ask it to as a follow-up prompt. You do need to have instructions on how to set up a test environment or use infra for which dev env testing is well-documented online.
And yes, it's non-deterministic. It's not gonna test everything exhaustively or systematically like a fuzzer. It's basically doing what a manual QA would, at best. It probably doesn't even cover every edge case each diff. But I'm fine with this tradeoff for not having unit tests.
Caveat emptor:
1. These are relatively small-medium programs (~10k LOC) where the frontend and backend code reside in the same repo, and the frontend is strictly a web app, and where 100% of the functionality can be tested in the dev environment (incl the database).
2. I'm the only person working on these repos right now.
3. I read every line written by the LLM and have things I don't like refactored or stashed and rewritten with a modified prompt, mostly based on intuition.
>How many tokens does this process take?
I genuinely believe anyone who exceeds usage on a $20 plan doesn't use Claude Code correctly. Always use a new context wherever possible. Always make surgical changes (or, broad changes with well-defined success criteria). For anything complex, get a plan from the LLM at least so that you know it understood your ask. Ctrl+C and `git stash` are your best friends. Keep everything clean, don't build castles on garbage. In the LLM era ironically you should never put off tech debt because it will just compound.
I take a more "hybrid" approach, myself, but I also have control issues. I am slowly moving towards all-LLM, but I'm not quite there, yet.
Have you ever worked on existing code base that has components that makes no sense but must be there?
What?!
Unit testing was to ensure the code operated according to an intended specification or outcome especially when it deals with a complex support matrix. It extended well beyond the codebase itself! You can have language errors where output behaviors differ between one supported version to another (ex. Java 11 vs Java 17), runtime errors that are specific to operating system quirks (ex. RHEL 8 vs RHEL 9), or workarounds added to your codebase for known bugs in different dependency libraries you support at compile/run time (ex. Net-SNMP 5.2 vs 5.5). It wouldn’t matter if someone read all your code; you would still have the potential for errors caused by the components needed to run your software.
If your project doesn’t have to worry about these types of issues then lucky you and continue on without unit tests for as long as you can.
The more types of potential errors your compiler catches, the fewer unit tests you need. Haskell needs fewer (and potentially different) tests than Perl or Python.
The other thing where unit tests really shine is testing a complex algorithm. If you think of a problem with a very well-defined output (like sorting, searching, some types of computation, graph algorithms, geometry problems etc.), then it's pretty easy to come up with a stable API and thus stable tests. I'm pretty convinced that unit tests give you more bang for the buck than plain-English test plans that you need to spend tokens on to evaluate.
I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them. It’s better to spend more time on the core implementation.
I respectfully disagree. IME writing a good test suite is one of the areas where current LLMs perform worst. They certainly like to write plenty of tests but those tests aren't necessarily adding much value and they don't necessarily cover the important functionality.
Given how heavily LLMs and specifically agentic workflows still rely on producing something vaguely correct and then iteratively applying guard rails to reach something that actually works I humbly suggest that trusting a very substandard test suite to be one of those guard rails might be unwise.
I realise this is heresy, but my conclusion is that unit tests aren't important any more. E2E testing on the other hand...
(I'm agreeing with you about the quality of LLM tests, I'm just trying to make lemonade out of all these LLMons).
All those extra tests are churning through tokens and CI resources. These are far from free in a direct financial sense. I've also noticed some of the more recent models will try to do a lot of experimentation and ad-hoc testing by spinning up shells or running something like Python scripts direct from a command line. Every time they do that you either have to review whatever they propose to run carefully - which is slow and tedious work - or stick the tools in YOLO mode and hope that your agents don't accidentally wipe your entire filesystem, open your firewall for malware to take over your network, and perhaps launch a few nuclear missiles.
I would respectfully disagree about the value of unit tests as well. The reason unit tests appear to have less value in the LLM/agent era is because current LLMs suck at decomposition and other architecture-level work. They just see a mass of code and change it as a mass of code. This probably results in orders of magnitude more code being written/changed than would have been necessary for a skilled developer working on a well-maintained code base to solve the original problem and that automated work also takes longer in wall time. Again this gets brushed under the carpet because a lot of people trust that LLMs are so good that their costs are negligible regardless of how much work they do. People used to say that about the Cloud transition as well because not needing so many in-house staff to manage your resources was going to be a huge saving. Today we have companies bringing in consultants because they literally can't even work out how much they're spending on AWS by themselves. Instead of having a few greybeards who could make a Linux server do anything we have a few increasinglygreybeards who know which combination of 17 AWS resources they need Terraform to spin up so they can write the web server equivalent of hello-world.c. Meet the new boss - same as the old boss. :)
I have an assumption that inference tokens are only going to get cheaper, I admit. We could get at least 10x efficiency savings by burning current models to ASICs. But today they're cheap enough to let the machine churn - that's the environment I'm making decisions in.
> I would respectfully disagree about the value of unit tests as well.
I honestly don't think the LLM-generated tests are at all valuable in a human sense. I get the impression the end result is "fewer bugs" (but I've never tried to prove that). That's why I'm happy for the LLM to have a directory it can write its own tests in. It's a scratch space.
> this probably results in orders of magnitude more code being written/changed than would have been necessary for a skilled developer working on a well-maintained code base to solve the original problem
Nothing gets committed in my name that I don't understand. I'm one of the people LLMs are actually making slightly slower, because I insist on reviewing the code and working single-threaded.
This problem existed before AI and great when you caught it.
The name of the test should indicate what it's doing.
For unit tests, these should check only one thing, and the test can be structured into "assemble, act, assert" style tests.
On the implementation side: code which touches state is tricky. -- So, minimize the amount of code which touches state (or rather, shift as much code as you can to stateless code, which is easier to bring under test).
In systems where I've taken complex inputs and transformed it into complex outputs, I'd make it as easy to bring each particular pass under test, have snapshots that the same transformations resulted in the same outputs (insofar as I did want that consistency).
I'd think with a compiler, though, it can still be practical to describe the input you depend on as a value.
In distributed systems, your service might rely on another service which might be broken, might return different values than you expect, might return slower than you expect, etc. and so testing the parts of your system which depend on other services can be tricky.
Point is, you can arrange a lot of your code into "it's easy for me to ensure this part works", and you can see wins from that, even if as a whole the full system can be difficult to bring under test.