Ask HN: When can we trust AI in coding, and avoid the happy path issue? | Hacker News Reader

Ask HN: When can we trust AI in coding, and avoid the happy path issue?

2 pointsabalgir7 comments

I have a question, and I hope you can clarify it. Consider this scenario: Claude code runs on a desktop to develop a solution, and its directive requires Codex to test everything it does under a different directive. The Codex directive is to refute Claude's test results, maintain its own test ledger, and return PASS or NO PASS. Claude cannot negotiate; it has to fix the NO PASS issues. Even with such a strict environment, I am ending up with problems, and of course, when asking Claude, the answer is "This is on me; I did not do X and Y"; when I ask it whether Codex checked it, "Codex checked all but not this "...

So when can we trust the AI to do the full job, not just the happy path? Is there a better way to impose quality?

I feel that on large projects, even if I upload Claude directives to re-read, Claude or Codex (we can switch roles) tends to drift and miss things.

View on HN

I think we will never achieve this scenario... The art of building software is about slicing the problem into the smallest chunks; the AI has been designed to answer the questions. The new models each time are better at solving hard problems that are related to the essential complexity, but they are not aware of the accidental complexity.

This is where we can add the differential value by applying the extra context, in terms of guardrails, architecture, and making the project more consistent. AI is better and faster than us generating code, but without this direction... I like to say the AI amplifies us; if before the AI you were someone who could build scalable products consistently, the AI is a multiplier, but if you didn't have this capability before the AI, now you will produce more tangled code, amplifying the feeling of mess.

Whenever we discover how the human mind does it, and then we apply it to AI systems. Currently AI just has a surface level interface that makes it seem like it can think of all these things humans think about, i.e. is this going to work properly, is this a good idea given the objectives, does this achieve the wider goal etc., etc.

It's ultimately hard to articulate and hard to measure. We are only measuring very narrow tasks (even if the tasks are large, there is no test that is measuring how well the AI took initiative or was able to solve unforeseen problems).

People like Yann LeCun understand this and are trying to build more wholistic level intelligence that would achieve this, while the majority of the current AI industry is closing their eyes and betting on scale and hoping that with enough compute these properties will emerge. I think we're going to be stuck with "dumb intelligence" until those other breakthroughs happen.

Don't you think we should build infrastructure around the AI with a watcher (yes, another AI) that makes sure AIs don't deviate or drift? I.e., we can ask the dev AI to submit its test ledger as a contract, the reviewer AI to submit its own as another contract, and the watcher to monitor contract executions for deviations.

Then I see what you might say: Who monitors the watcher? -:(

Exactly. We can sure try, but I would bet against it working well. It might work a little, maybe it would be worth it, but the underlying assumption is that AI can have good judgement and so far that is mostly an illusion and it's closer to random. Sometimes it makes good decisions, sometimes not, and that is a huge problem that keeps it from being good agentic systems. They still need the harness and all this other traditional code around it to get it to work.

Totally agree, they need a strong harness to make them work

Never?

Testing has always been a game of figuring out how much is enough to produce a stable product, without burning massive amounts of engineering time with testing. LLMs have not changed that. They will never be perfect, and that is OK. Same as with humans, you need to define your own comfort zone for quality controls and accept that there will be bugs. Fixing those bugs will be part of your process. If you get too many, increase testing. If you get none, you just have not yet discovered what was missed.

I like to accept the concept that even with AI, we will have bugs, and we need to live with this reality.

Reply on news.ycombinator.com