Hacker News Reader

Comment by desolate_muffin | Hacker News Reader

I haven’t had this happen yet, but I was just thinking about this last night. I would be very interested to see if exercising TDD, as difficult as that practice is to actually do, would have a meaningful positive impact on improving this phenomenon. Usually agents model what most humans do, writing tests to pass and assert that the code you just wrote works as you wrote it.

View on HN

It has the same failure mode as dealing with TDD by a human engineer, which includes things like mocking the function under test in the test and then testing the mock, asserting only things that aren't actually exercised by the function under test (like asserting the initial user ID is the same as the final user ID for a function that changes name of a group), not validating side effect behavior (or lack thereof) for a function that's only purpose is the side effect (like appending a log line to a log file), and more!

What changes with TDD is that the important part of the review moves from the mainline code to all the TDD tests and the coverage, and you also need to see a test run from before the code changes with all the new/updated TDD tests failing in addition to just a test run after the changes with all the (at least) new/updated TDD tests passing.

This is part of the reason that I've come to view mocks as a code smell. They're too easy to screw up.

But even more than that, I've found that modules that need to be tested with mocking can often be reorganized to be testable without mocking. And doing so tends to also make the code easier to understand and easier to modify without breaking things in the future.

Mocks/stubs definitely have a smell to them, but especially if you want to do more than just unit tests can be necessary. I've found that setting up the mainline architecture so that components that are allowed to be mocked have a single access path in the code makes it easy to mock at a level low enough that the only thing being mocked is that component and then preemptively building out those mocks into the test framework works decently well. For me, 3rd party services are usually fine to mock, but things like DBs and redis are usually not. I'd rather spend the time to make a test container the devs can run locally so that they can't skip over checks/constraints and the integrations with services/components we have full control over.

That way you can tell at a glance of the test file change just how bad it is. Seeing new raw mocks (eg sinon in the test file at all) instead of the existing mocks is an instant no, and seeing an existing mock getting added requires at least investigation for necessity.

It does. But keep an eye on whether the tests actually run.

I’ve had decent luck with it as long as I at least stub out the test cases with a test name and a “given … when … then” comment.

Otherwise it seems the AI has a tendency to write the worst kind of TDD antipattern microtests that simultaneously make refactoring impossible while failing to document or even confirm the most important functional requirements.

Once it’s got those stubs it does do a pretty good job of suggesting behaviors I forgot to specify, and other improvements. It’s just “designing” test fixtures from the ground up that seems to put it into “generate plausible-looking slop” mode.

Reply on news.ycombinator.com