Hacker News Reader

Comment by jacquesm | Hacker News Reader

Without all that slop how would you have trained the AI?

What makes you think that AI trained on slop is suddenly going to produce quality code?

I think ai is more rigorous with testing. It’s not so much about what an llm “knows”, but more about how well it does plan-do-check-act.

AI is as good as the prompt. It's a simple as that. If you don't foresee everything and anything that can go wrong AI isn't not going to magically fill in the blanks and save you. So in projects that already had good test discipline, well written specs and tickets and very clear expectations, coding guidelines and documentation it is relatively easy to fit an AI in for guided work. You still need to check up but chances are good that it will usually work as expected.

But any kind of mistake, any kind of ambiguity and you'll be in for a wild ride. It usually takes a while for the rot to set in. I've had to revert - more than once - to many commits prior to undo a bunch of work that had me completely puzzled as to what the reason that particular fork was taken to begin with. Usually I end up figuring it out but definitely not always. In some cases this kind of wild modification fed back on itself until the situation was beyond rescue. I'm a bit quicker with such reverts now than in the past, for the first year or so I would try to save what could be saved, now I just restore and try again.

But if you're not sharp when evaluating what gets done there is a good chance that you'll miss it and then it can pop up way later.

What I've noticed as correlation (but I'm not sure if that is the root cause) is that non-local effects and stack effects are particularly hard as is everything that requires a working model of the world.

edit: this comment seems to make people uncomfortable.

I disagree that the prompt is the most important thing, I'd say other things like the model you use, a clean codebase, actually reading code and using discretion about which changes are accepted are far more important if you're using AI for work that needs to be correct. LLMs still make significant mistakes in design especially.

Harness, prompt etc etc are all peripheral IMO, though people like to think their super long prompt, custom harness and MakeNoMistakes.md skills are what contributes to their special skill with AIs, but really models are just improving.

You can just trust the AI to generate a perfect answer I suppose (and why are humans slowing it down at all if so?), but if you don't expect that yet you have to put in the work.

I think I noticed claude uses the shape and style of the existing code base to generate the new code. When reviewing claude's output, it looks familiar and the patterns are similar to what I would write.

I'm not sure who is studying this yet, but there has to be a statistical relationship between three factors of

1. intelligence of the human: the design patterns and architecture is already modelled in the mind, steering the AI becomes easy to know what to say next

2. quality of the codebase: the LLM can pick up the adjacent patterns and write code that looks and works cleanly.

3. quality of the prompt: the prompt keeps the AI focused and ignores the trivial side effects; keep the AI wandering away from critical path work; sometimes the AI can't make the decision what call site is next

In summation, what if claude is just a mirror pointing back at us? It certainly would explain wildly divergent opinions of opus 5.5

There are definitely nuances, and yes of course the prompt must make some difference, but I'm not convinced it makes as big a difference as some think, particularly a long prompt or complex setup of harnesses etc surrounding the LLM model.

To me the order of importance would be:

1. The model - if you use a terrible model you'll get bad output

2. The codebase - this definitely forms the model for what claude outputs to some extent.

3. The human operator, what they accept and reject etc.

4. Things like prompts, skills, harnesses etc. The quality of the prompt.

So I think our positions are really just a different ordering of what is important rather than fundamentally opposed.

Re Claude being a mirror, if it is it's a particularly distorting mirror which often changes things in subtle ways and is determined to make people feel smart and addicted to its output - it has been trained to do so after all. I really don't think a mirror is the best analogy, because it's too independent and seductive to be a mirror.

There's a good example of the sort of thing they produce without good guidance as a dead comment on this thread as a reply to yours. Encouraging, positive and superficially plausible, but without any actual meaning or intention. The equivalent of that in code is what we need to be wary of.

I think the environment you set up for the agents is just as important as the prompt. I've been working full time on a claude code developed project for the last 10 months, and it seems to become better and better at developing the right thing. It has the tools to research whatever context it needs, test it's output in a relevant way, etc. It doesn't by itself suggest complete refactoring, or big performance optimizations, so that's something you have to prompt for, but it will collect all the evidence and context to make a proposal, implement it, and test it.

Every single case of AI slop code I've encountered in a shipped product is a result of a complete lack of proper, professional workflow on the part of the human who decided to ship it. People are using AI as a reason to continue to justify their lack of responsibility for what it is they're doing.

And on the other side, I have seen AI produce exceptionally efficient, productive and effective code - the opposite of slop - when it is treated like a junior developer who will do any idiot thing, except when it is mentored by an actually experienced, professional developer.

The source of slop in AI code is not AI, its not the training data - its in the actual workflow of the human involved in its production.

Yes, that I can agree with.

If you just turn it loose on a codebase and ship it when it passes the tests you're asking for getting burned.

The most effective use of AI is in an environment which already produces high quality software through proven tooling and methodology. If your workflow is broken, your AI isn't going to fix that - except by way of giving you a lot of slop to push through whatever pipes you're already not getting flow through.

What's your workflow, out of interest?

Two tracks: code I don't care about where AI can be used and core stuff that I do care about and that needs to be production grade (real time stuff). The core stuff I write myself, the auxiliary stuff is almost all automated. This doesn't get me to 'maximum speed' but it means I keep total control over what I produce and AI can review it, but read only, no changes and it is set up just to flag things, not to suggest particular fixes.

>two tracks .. review

Good one, that seems about right by my read of things.

Review is more important now than ever. My workflow has 3 review stages .. after analysis/specs/design, just prior to starting development (intake), and in the quality stages of testing and so on.

Well, there's also an after-release review, but the sales/marketing guys do that as part of their followups.

The code AI generates is fine, it's the context on a codebase that's often missing. Generating duplicate code because they didn't see that a Class already existed for the same thing. Or having no clue that changing a method will fix A but often missing that it will break B.

Reply on news.ycombinator.com