Hacker News Reader

I simply test the final result,manual tests, automated tests, benchmarks, people using the apps and giving feedback, increased number of sales, etc.

View on HN

Here is something I found this week: copies the entire db to memory, inserts a single row into in-memory db, deletes everything from real db, writes everything back. Sentence that caused this is "do XXX in one transaction".

All tests pass, everything seemed to work. Would not have noticed of I didn't read through the code. Amount of other stupid shit I found is crazy.

These anecdotes mean little without knowing context, model, harness, prompt. The technology has moved a lot in the past year, and does require some adjustment.

My very first experience with OpenCode led to a `rm -rf /` within minutes, which I caught by chance watching the output stream in. Yet 99% of my code is currently written through an agent.

Ah yes, the favorite response to any criticism of LLMs – you are using the wrong model.

A performance benchmark would have easily caught this though, ask the LLM to create and run benchmarks and see how those change after each new feature/major change.

I have found test coverage reports to be really helpful as well.

Reply on news.ycombinator.com