Hacker News Reader

Comment by xyzsparetimexyz | Hacker News Reader

If you don't look at the code, how do you know it's higher quality than ever?

I simply test the final result,manual tests, automated tests, benchmarks, people using the apps and giving feedback, increased number of sales, etc.

Here is something I found this week: copies the entire db to memory, inserts a single row into in-memory db, deletes everything from real db, writes everything back. Sentence that caused this is "do XXX in one transaction".

All tests pass, everything seemed to work. Would not have noticed of I didn't read through the code. Amount of other stupid shit I found is crazy.

These anecdotes mean little without knowing context, model, harness, prompt. The technology has moved a lot in the past year, and does require some adjustment.

My very first experience with OpenCode led to a `rm -rf /` within minutes, which I caught by chance watching the output stream in. Yet 99% of my code is currently written through an agent.

Ah yes, the favorite response to any criticism of LLMs – you are using the wrong model.

A performance benchmark would have easily caught this though, ask the LLM to create and run benchmarks and see how those change after each new feature/major change.

I have found test coverage reports to be really helpful as well.

Also to make it clear, I am not saying the code itself is better, but the end-product and what the user sees and gets. Like that saying "if it's stupid, but it works, then it's not stupid"

for you, "if it's stupid, but it works, then it's not stupid."

for me, if it's stupid, but it works, then it's a coincidence.

I think LLMs themselves are a good example of "it's stupid but it works"

Because the exact same processes that create it faster, are also now generating large comprehensive test suites, and data-validation suites based around the actual business requirements and data/edge cases.

In essence, it is writing the code faster (great), but then also writing the code that was often previously never written due to budget/time limitations (comprehensive testing/validation).

So, how do you know the code is the highest quality ever? Because we have test result data that PROVES it to be the case. It isn't "trust but verify" it is "Don't trust. Verify."

How do you know those large comprehensive test and validation suites you've never looked at actually reflect real user experiences at all?

You must have replied to the wrong comment, since the one you did replied to never claimed that the test and validation suites were "never looked at." They are reviewed, and the test/data results are reviewed.

It said verbatim: "Don't trust. Verify."

> Because we have test result data that PROVES it to be the case.

pgrust thought this too, but then another human discovered claude had hardcoded benchmark query plans.

Now pgrust has a "test" [1] to make sure the Ai doesn't do this again (unless it decides this test is bad for the current goal)

I have a growing list of these, another notable one is OpenCode having a security regression [2] because the Ai redesigned configuration loading for v2.

Real verification seems beyond transformer capabilities.

[1] https://github.com/malisper/pgrust/blob/38634b408dea373f3944...

[2] https://github.com/anomalyco/opencode/pull/54921

Yeah but you don't test for stuff you don't know you have to test for. All tests can pass and internally its still done in a braindead way that will explode at some point.

Absolutely, and this is exactly why humans are weaker than LLMs. We consider what is likely, they consider what the data/logic shows is possible. So you get edge cases tests/flagged, that wouldn't even be something a real person would know to look for.

This is my observation as well. I thought I was good at testing (I wanted to be good at testing), but the models are just better at it. Testing in particular rewards a tireless cataloging of the tiny and the tedious and in this respect LLMs are unmatched.

Single data point, my Sentry notifications across more than a half dozen client projects when from 2-4 a month to basically 0. That's while shipping 20x _more_ code.

Very little code written in the world day to day ever has to justify its 'quality' or long term value. A lot it is just one off tools, bits and bobs to make a teams life easier or whatever.

I used to work in Web agency for sizeable brands and not a single site I built lasted over a year because the companies changed concept regularly. The next agency would come in and tell them it was built wrong and couldn't be reused etc etc. We did the same, it was irrelevant how it was built and we didn't even look. This is the nature of most code. The problem is that very few people will ever be in a position to be writing verifiable important code, but every developer likes to cosplay that they are.

Most code is ultimately an, annoyance businesses need to meet a goal that will always be a cost. It's often worth treating it as such and writing things that are easy to spin up and teardown.

The people most vexed by AI dev appear to be the ones who think writing code is a pinnacle role, it's not even though over our lives the salaries would sometime suggest otherwise. That's ultimately going to be a short term blip in the landscape and most code will be one seen as the cost it is.

How do you know it's high quality if you do look at the code?

A programmer's output was never code. It's was just what we had to do to get computers to work for us. Now I just ask a genie to do what I did and check the results.

Good for you. I prefer to communicate with computers using an unambiguous language that describes exactly what I want them to do.

Ah, a connoisseur of machine code.

Competitive programmers don't usually produce quality code, I'm guessing it could hardly be worse even without looking at it.

That's true, code is usually shit but the result/outcome really good and performant.

They do produce code optimizing for a different goal. But readability is often not one of them yeah.

He must have been a really bad coder.

Claude told him so.

Reply on news.ycombinator.com