Hacker News Reader

That suggests a test suite to test the test suite, which again would suggest another test suite to test the test suite testing test suite...

In the end, it is test suites all the way down.

View on HN

There actually is a way to test your test suite: mutation tests [1]. Basically, you change the covered codebase (invert an if statement, change a variable, etc.) and expect the tests to then fail. If the tests actually survive the mutations, they might not be good enough.

[1] https://en.wikipedia.org/wiki/Mutation_testing

You could evaluate the quality of your test suite with something other than a test suite.

such as?

A developer who knows the code and will own the consequences can review and merge it - easy. Just not sure why the LLM needed to get involved in the first place.

Presumably because they've got 500 test cases?

Well, you need someone to write the code, and someone (else) to review the code.

In this situation you would replace the author with the LLM, but leave the reviewer as human.

It's not as pointless as you make it out to be. You still save one human.

You can run mutation tests that intentionally seed errors into the codebase (flipping Boolean and to or, for example) and rerun the tests.

A good test suite will catch these errors. Something that is effectively a noop to get to green will not.

Reply on news.ycombinator.com