Hacker News Reader

Comment by roncesvalles | Hacker News Reader

I take the complete opposite approach which is to have zero unit tests now, because LLMs are excellent at both reviewing code to understand what's going on, and also doing manual QA given environment instructions. Lots of unit tests will just bloat the context which ultimately dilutes the quality of the LLM's work and also creates additional work (it has to update the code and the tests each time).

Unit testing imo is pre-LLM. It was mostly to keep the human honest because the human can't possibly read all the code to make sure nothing broke.

View on HN

I'm flabbergasted at this comment.

Are you maintaining a list of all invariants you want QA'd by the LLM after each change? How many tokens does this process take?

right? I love HN. This is the new hot take, spoken as confidently as possible -- NO TESTS AT ALL.

You don't need tests. Your users are going to do all the testing!

The README defines the expected behavior of my API endpoints and the overall program in natural language. Some repos have additional documents (like DATA_MODEL.md) that defines the nitty gritty of the distributed concurrency characteristics, data invariants and such other things.

Claude Code (Opus 5.5 xhigh), when it makes changes, tests the relevant endpoints for behavior based on what it touched. You could also just explicitly ask it to as a follow-up prompt. You do need to have instructions on how to set up a test environment or use infra for which dev env testing is well-documented online.

And yes, it's non-deterministic. It's not gonna test everything exhaustively or systematically like a fuzzer. It's basically doing what a manual QA would, at best. It probably doesn't even cover every edge case each diff. But I'm fine with this tradeoff for not having unit tests.

Caveat emptor:

1. These are relatively small-medium programs (~10k LOC) where the frontend and backend code reside in the same repo, and the frontend is strictly a web app, and where 100% of the functionality can be tested in the dev environment (incl the database).

2. I'm the only person working on these repos right now.

3. I read every line written by the LLM and have things I don't like refactored or stashed and rewritten with a modified prompt, mostly based on intuition.

>How many tokens does this process take?

I genuinely believe anyone who exceeds usage on a $20 plan doesn't use Claude Code correctly. Always use a new context wherever possible. Always make surgical changes (or, broad changes with well-defined success criteria). For anything complex, get a plan from the LLM at least so that you know it understood your ask. Ctrl+C and `git stash` are your best friends. Keep everything clean, don't build castles on garbage. In the LLM era ironically you should never put off tech debt because it will just compound.

This sounds a sensible approach. I have many of the same features, in my development.

I take a more "hybrid" approach, myself, but I also have control issues. I am slowly moving towards all-LLM, but I'm not quite there, yet.

Ok, but why not just have the LLM generate tests? They're cheaper to run and more repeatable.

> LLMs are excellent at both reviewing code to understand what's going on ...

Have you ever worked on existing code base that has components that makes no sense but must be there?

I think it really depends on the language and the project.

The more types of potential errors your compiler catches, the fewer unit tests you need. Haskell needs fewer (and potentially different) tests than Perl or Python.

The other thing where unit tests really shine is testing a complex algorithm. If you think of a problem with a very well-defined output (like sorting, searching, some types of computation, graph algorithms, geometry problems etc.), then it's pretty easy to come up with a stable API and thus stable tests. I'm pretty convinced that unit tests give you more bang for the buck than plain-English test plans that you need to spend tokens on to evaluate.

> Unit testing imo is pre-LLM. It was mostly to keep the human honest because the human can't possibly read all the code to make sure nothing broke.

What?!

Unit testing was to ensure the code operated according to an intended specification or outcome especially when it deals with a complex support matrix. It extended well beyond the codebase itself! You can have language errors where output behaviors differ between one supported version to another (ex. Java 11 vs Java 17), runtime errors that are specific to operating system quirks (ex. RHEL 8 vs RHEL 9), or workarounds added to your codebase for known bugs in different dependency libraries you support at compile/run time (ex. Net-SNMP 5.2 vs 5.5). It wouldn’t matter if someone read all your code; you would still have the potential for errors caused by the components needed to run your software.

If your project doesn’t have to worry about these types of issues then lucky you and continue on without unit tests for as long as you can.

My name. Your fault.

Reply on news.ycombinator.com