Lots and lots of testing. I usually interactively test with the LLM. Now that I'm spending less time, coding, I get to spend more time, testing.
In my experience, I test more, with an LLM, than I did, with my own code. I don't find coding errors, but I do find implementation errors.
On another note: I ran an LLM on a bunch of really mature code that I wrote, and could swear was "perfect," and it found a number of issues.
Nobody's perfect. Trust; but verify.
What have we gained aside from shifting where time and effort is spent in the design loop?
- code? up - build budget? Up - developer output? Up - developer oprrational costs? Waaay up!
Why is this Bad? Every one involved profits somehow.
The verification is what we always wanted but never had time for. For 30 years I wrote side project games without extensive testing or a fuzzer or anything because I wanted to spend my time writing the game. Now I can do that and still have the testing I always knew I should be building.
Right now, we're in a "honeymoon" stage, where non-engineers think that they can somehow, magically produce stuff that is comparable to what experienced and disciplined engineers can do.
That won't last long.
Despite all the caterwaulin', LLMs are just another engineering tool, and good engineers will figure out how to implement them, in ways that absolutely blow away the garbage produced by non-engineers.
It's like that rich, no-talent guy that buys a PRS guitar, and thinks he'll beat the experienced musician that still uses his beat-up old strat, but consistently makes hits.
In my optimistic days, I think we'll start putting mechanisms in place that will have LLMs help develop cognition, skills and expertise. It'll usher in an era of mastery and wonder.
On my bad days, I despair that the siren song of "good enough" slop will overwhelm even the most disciplined minds, and we're facing the atrophy of civilization.
Will this just end up exaggerating the pereto distribution? Will there end up being this tiny percentage of people that are driven by natural curiosity to do the very hard work of really understanding hard things? Will society even have any sort of reward for those people?
This method also helps with the classic problem of being unable to identify essential complexity up front.
This is why LLM coding fails.
This truly is the worst thread on HN. Now we're down to explaining 102 topics. Geez guys, what did we gain from evading regression errors? Who knows!
If someone finds it burdensome to review my 50-line AI-generated bug fix, adding 200 lines of tests only gives them more lines to review.
I have no shame in accepting vast quantities of ai generated automated tests at review time without really paying attention to them. It’s better to spend more time on the core implementation.
I respectfully disagree. IME writing a good test suite is one of the areas where current LLMs perform worst. They certainly like to write plenty of tests but those tests aren't necessarily adding much value and they don't necessarily cover the important functionality.
Given how heavily LLMs and specifically agentic workflows still rely on producing something vaguely correct and then iteratively applying guard rails to reach something that actually works I humbly suggest that trusting a very substandard test suite to be one of those guard rails might be unwise.
I realise this is heresy, but my conclusion is that unit tests aren't important any more. E2E testing on the other hand...
(I'm agreeing with you about the quality of LLM tests, I'm just trying to make lemonade out of all these LLMons).
All those extra tests are churning through tokens and CI resources. These are far from free in a direct financial sense. I've also noticed some of the more recent models will try to do a lot of experimentation and ad-hoc testing by spinning up shells or running something like Python scripts direct from a command line. Every time they do that you either have to review whatever they propose to run carefully - which is slow and tedious work - or stick the tools in YOLO mode and hope that your agents don't accidentally wipe your entire filesystem, open your firewall for malware to take over your network, and perhaps launch a few nuclear missiles.
I would respectfully disagree about the value of unit tests as well. The reason unit tests appear to have less value in the LLM/agent era is because current LLMs suck at decomposition and other architecture-level work. They just see a mass of code and change it as a mass of code. This probably results in orders of magnitude more code being written/changed than would have been necessary for a skilled developer working on a well-maintained code base to solve the original problem and that automated work also takes longer in wall time. Again this gets brushed under the carpet because a lot of people trust that LLMs are so good that their costs are negligible regardless of how much work they do. People used to say that about the Cloud transition as well because not needing so many in-house staff to manage your resources was going to be a huge saving. Today we have companies bringing in consultants because they literally can't even work out how much they're spending on AWS by themselves. Instead of having a few greybeards who could make a Linux server do anything we have a few increasinglygreybeards who know which combination of 17 AWS resources they need Terraform to spin up so they can write the web server equivalent of hello-world.c. Meet the new boss - same as the old boss. :)
I have an assumption that inference tokens are only going to get cheaper, I admit. We could get at least 10x efficiency savings by burning current models to ASICs. But today they're cheap enough to let the machine churn - that's the environment I'm making decisions in.
> I would respectfully disagree about the value of unit tests as well.
I honestly don't think the LLM-generated tests are at all valuable in a human sense. I get the impression the end result is "fewer bugs" (but I've never tried to prove that). That's why I'm happy for the LLM to have a directory it can write its own tests in. It's a scratch space.
> this probably results in orders of magnitude more code being written/changed than would have been necessary for a skilled developer working on a well-maintained code base to solve the original problem
Nothing gets committed in my name that I don't understand. I'm one of the people LLMs are actually making slightly slower, because I insist on reviewing the code and working single-threaded.
This problem existed before AI and great when you caught it.
The name of the test should indicate what it's doing.
For unit tests, these should check only one thing, and the test can be structured into "assemble, act, assert" style tests.
On the implementation side: code which touches state is tricky. -- So, minimize the amount of code which touches state (or rather, shift as much code as you can to stateless code, which is easier to bring under test).
In systems where I've taken complex inputs and transformed it into complex outputs, I'd make it as easy to bring each particular pass under test, have snapshots that the same transformations resulted in the same outputs (insofar as I did want that consistency).
I'd think with a compiler, though, it can still be practical to describe the input you depend on as a value.
In distributed systems, your service might rely on another service which might be broken, might return different values than you expect, might return slower than you expect, etc. and so testing the parts of your system which depend on other services can be tricky.
Point is, you can arrange a lot of your code into "it's easy for me to ensure this part works", and you can see wins from that, even if as a whole the full system can be difficult to bring under test.
Unit testing imo is pre-LLM. It was mostly to keep the human honest because the human can't possibly read all the code to make sure nothing broke.
Are you maintaining a list of all invariants you want QA'd by the LLM after each change? How many tokens does this process take?
Claude Code (Opus 5.5 xhigh), when it makes changes, tests the relevant endpoints for behavior based on what it touched. You could also just explicitly ask it to as a follow-up prompt. You do need to have instructions on how to set up a test environment or use infra for which dev env testing is well-documented online.
And yes, it's non-deterministic. It's not gonna test everything exhaustively or systematically like a fuzzer. It's basically doing what a manual QA would, at best. It probably doesn't even cover every edge case each diff. But I'm fine with this tradeoff for not having unit tests.
Caveat emptor:
1. These are relatively small-medium programs (~10k LOC) where the frontend and backend code reside in the same repo, and the frontend is strictly a web app, and where 100% of the functionality can be tested in the dev environment (incl the database).
2. I'm the only person working on these repos right now.
3. I read every line written by the LLM and have things I don't like refactored or stashed and rewritten with a modified prompt, mostly based on intuition.
>How many tokens does this process take?
I genuinely believe anyone who exceeds usage on a $20 plan doesn't use Claude Code correctly. Always use a new context wherever possible. Always make surgical changes (or, broad changes with well-defined success criteria). For anything complex, get a plan from the LLM at least so that you know it understood your ask. Ctrl+C and `git stash` are your best friends. Keep everything clean, don't build castles on garbage. In the LLM era ironically you should never put off tech debt because it will just compound.
I take a more "hybrid" approach, myself, but I also have control issues. I am slowly moving towards all-LLM, but I'm not quite there, yet.
Have you ever worked on existing code base that has components that makes no sense but must be there?
What?!
Unit testing was to ensure the code operated according to an intended specification or outcome especially when it deals with a complex support matrix. It extended well beyond the codebase itself! You can have language errors where output behaviors differ between one supported version to another (ex. Java 11 vs Java 17), runtime errors that are specific to operating system quirks (ex. RHEL 8 vs RHEL 9), or workarounds added to your codebase for known bugs in different dependency libraries you support at compile/run time (ex. Net-SNMP 5.2 vs 5.5). It wouldn’t matter if someone read all your code; you would still have the potential for errors caused by the components needed to run your software.
If your project doesn’t have to worry about these types of issues then lucky you and continue on without unit tests for as long as you can.
The more types of potential errors your compiler catches, the fewer unit tests you need. Haskell needs fewer (and potentially different) tests than Perl or Python.
The other thing where unit tests really shine is testing a complex algorithm. If you think of a problem with a very well-defined output (like sorting, searching, some types of computation, graph algorithms, geometry problems etc.), then it's pretty easy to come up with a stable API and thus stable tests. I'm pretty convinced that unit tests give you more bang for the buck than plain-English test plans that you need to spend tokens on to evaluate.
Even if your test suite tested every single possible behavior and code path, tests usually can't intercept all possible unintended behavior that an LLM might have added.
In all likelihood, most test suites don't intercept any unintented behavior (such as unintended internet connections).
That's not a big problem when a honest human has written or reviewed the code, because unintended code is usually easy to spot.
But an LLM might have dropped any whatsoever code: maybe some of your project's code resembles the one of a trojan in its training, so it completes it with the trojan.
Or maybe it found that if your server is unreachable, it can upload the client's data to a public Github repo; or that if there's no space left on the device, it can delete some of the user's folders, easy!
Would your clients be understanding if that happened?
Eventually, the tools become trustworthy, and we no longer need to worry about their products.
We're just about there, with frontier LLMs, but I don't think we're all the way there, yet.
English isn't a good programming language if you want to trust that the output will be as you want it to be. High level code has no room for ambiguity - that's why we use it. It's a concise, very explicit language to tell the compiler exactly what to do.
Dealing with an LLM is a fundamentally different thing. The reason it's able to speed us up is because it makes assumptions for us and reads between the lines. The only world where we trust the output is one where it can literally read our minds. That's not coming any time soon
The human developer+compiler+linker toolchain isn't free of bugs either, some percentage it will produce code that fails loudly and sometimes it will produce code that has silent bugs, be it due to wrong source code or due to compiler or linker bugs.
LLMs instead are inscrutable, can output whenever at any time and you can't "fix their bugs"!
And don't attempt to place the probability of error of LLMs and normal software on the same plane, LLMs for the time being are wildly more unpredictable.
Making them review LLM-generated code will probably catch some errors and vulnerabilities, but could well leave there insanely dangerous hallucinated code.
Meanwhile I've been dealing with so much human slop (including mine) for the last 25 years that what comes out of LLMs in 2026 is an absolute relief.
You don't see what's going to happen if you don't even have a look at it?
I feel like I'm in some candid camera
Current LLM output is so much better than what I or my colleagues used to write, on every levels. Better readability when properly steered, better performance, better security, everything. Frankly this year has been a humbling experience.
> Do I live on a different universe ?
I guess...
> You don't see what's going to happen if you don't even have a look at it?
Who said we don't even look at it ?
You are aware that howwer much better to your hellish experience the average LLM output is, it's going to spit brain-dead, extremely dangerous nonsense sometimes?
Again, in my experience it's very rare (and rarer than at the beginning of this year). What was your experience with these tools ?
I'm not OP, and I'm not defending the point of not looking at the code at all or not doing any review. I do think checking each line is definitely not necessary and the amount of verification should be adapted to your domain.
How can you verify the code if you don't verify "each line"?
You can maybe check them with less care, yes.
But even for the most frivolous domains, in most cases malicious software can cause a lot of damage (for on example on older systems), so it is irresponsible to not check it, in my opinion.
There'd be a debate to have about why LLMs are threatening to some engineers while they are empowering to others. I'm having a blast producing the best output I ever done while some of my colleagues are suffering to the point of thinking about changing careers. There must be deep psychological reasons to this.
For open source libraries there's a chance that some of its users reviewed the code, for commercial libraries there's the legal risks holding them from being excessively careless.
LLM code instead can only be looked upon by you.
And, until a short while ago we were worrying of how to reduce the risks placed by libraries, with dependency cooldowns etc. Now we decided that we can ship random code seen by absolutely no one?
> There'd be a debate to have about why LLMs are threatening to some engineers
You're claiming that if an engineer doesn't like LLMs it's because he feels threatened by them?
There definitely are at least two very different groups of software developers: one that enjoys writing code, and the mechanicality and predictability of it, and one that maybe doesn't love developing, and for sure cares more about making a product, and maybe about the money that you can earn from it.
This is the purely psychological difference, but then there's a group of developers who feels the responsibility for what they do, realizes what effects their software can have and does what it can to protect its users; and another group that doesn't care too much, and considers their user's fault to decide to use their software, and to possibly harm other people through its use.
And, if your capabilities are limited, or your company forces you to go too fast, for sure the average LLM output can be better than your average code.
Not all LLMs fans are like that, of course, many just have misconceptions and don't realize the risks.
All your points so far boil down to the idea that the average LLM output is worse than the average engineer output. Sorry, it's not.
I hope you're not a real engineer, or at least that you don't really think what you've been saying.
I feel like your suggestion would be a nice organic extension of that same practice (at the cost of more infrastructure/tokens)
The goal is to get to a point where I can say with a good level of confidence that a change didn't break anything. It might be the wrong change, and it might be crap code, but I can ship it without anything blowing up. I've been using it for a while across a few game projects and so far it's worked quite well.
Black-box e2e testing isn't suitable, or practical, for a lot of applications. Unit and integration tests generally require you to understand the code you want tests for, to know what to actually assert
Are you just trusting the AI to do it all?
Testing is NOT verification.
EDIT: Sorry, oops, it's also called a kind of verification.
But there is the problem that tests only check or verify a small sample of possible cases or inputs. They can never show that the program is correct. (Famous Dijkstra statement.) To know something is really correct you have to understand and prove it.
1. The reason OOP complained about AI code being shipped without comprehension is because they want to ship of it more and faster. If they had time, they'd review the code. Writing tests requires a lot more time than what people usually spend on code reviews (unfortunately, but that's how it is). And if you think you can generate tests, again, by using AI, then, the bad news are that now you have both the code you don't understand and the tests you don't understand. Don't forget that running tests also takes time, in some cases it takes longer than to write code. So, again, your benefits obtained from AI-generating code won't materialize.
2. Even if magically you were able to procure decent-quality tests (without understanding what you are testing!) There are still problems not addressed by testing s.a. reusability. If you don't understand what the code does, how can you possibly reuse it for something else? Perhaps a small modification or a generalization of the existing code would allow you to reuse the mostly tested codebase for the new feature, but since you don't know what it does, you've missed that opportunity, and, again, the AI-generated code came second in the race to ship the product faster (also, you've got a worse product).
* * *
The question of "how to deal with code we don't understand?" is by no means a new one. Dijkstra's structured programming was a response to precisely this question. His argument showed that programmers can't even anticipate all the potential problems to be found in a 16-bit adder and that exhaustive testing is prohibitively expensive. Therefore, he argued, we should restrict the kinds or programs we write to those that are easier to understand (and that's what he called "structured" programming).
Perhaps, we can find a similar sort of harness for AI writing code, where we can impose boundaries it cannot cross that ensure understanding. I have no idea what that can look like, but I'm absolutely sure that testing isn't it.