Hacker News Reader

Comment by Someone1234 | Hacker News Reader

Because the exact same processes that create it faster, are also now generating large comprehensive test suites, and data-validation suites based around the actual business requirements and data/edge cases.

In essence, it is writing the code faster (great), but then also writing the code that was often previously never written due to budget/time limitations (comprehensive testing/validation).

So, how do you know the code is the highest quality ever? Because we have test result data that PROVES it to be the case. It isn't "trust but verify" it is "Don't trust. Verify."

View on HN

This is my observation as well. I thought I was good at testing (I wanted to be good at testing), but the models are just better at it. Testing in particular rewards a tireless cataloging of the tiny and the tedious and in this respect LLMs are unmatched.

Single data point, my Sentry notifications across more than a half dozen client projects when from 2-4 a month to basically 0. That's while shipping 20x _more_ code.

Yeah but you don't test for stuff you don't know you have to test for. All tests can pass and internally its still done in a braindead way that will explode at some point.

Absolutely, and this is exactly why humans are weaker than LLMs. We consider what is likely, they consider what the data/logic shows is possible. So you get edge cases tests/flagged, that wouldn't even be something a real person would know to look for.

> Because we have test result data that PROVES it to be the case.

pgrust thought this too, but then another human discovered claude had hardcoded benchmark query plans.

Now pgrust has a "test" [1] to make sure the Ai doesn't do this again (unless it decides this test is bad for the current goal)

I have a growing list of these, another notable one is OpenCode having a security regression [2] because the Ai redesigned configuration loading for v2.

Real verification seems beyond transformer capabilities.

[1] https://github.com/malisper/pgrust/blob/38634b408dea373f3944...

[2] https://github.com/anomalyco/opencode/pull/54921

How do you know those large comprehensive test and validation suites you've never looked at actually reflect real user experiences at all?

You must have replied to the wrong comment, since the one you did replied to never claimed that the test and validation suites were "never looked at." They are reviewed, and the test/data results are reviewed.

It said verbatim: "Don't trust. Verify."

Reply on news.ycombinator.com