abalgir

I have found that building a strict pipeline (custom dev) around the agents helps keep them honest and focused. But it's complex development; the agent model and effort don't guarantee good results. For now, having multiple agents from different companies check each other and the surrounding code is the best way to ensure excellent results.

Totally agree, they need a strong harness to make them work

I like to accept the concept that even with AI, we will have bugs, and we need to live with this reality.

Don't you think we should build infrastructure around the AI with a watcher (yes, another AI) that makes sure AIs don't deviate or drift? I.e., we can ask the dev AI to submit its test ledger as a contract, the reviewer AI to submit its own as another contract, and the watcher to monitor contract executions for deviations.

Then I see what you might say: Who monitors the watcher? -:(

I am working on a development harness using AI agents. There are plenty on the market, but I wanted something that encourages builders to build commercial solutions, not small POCs. In my experience, most stop at demos and betas. The name is funny, I know, but it has a significant meaning for me. When I started using AI, even with excellent prompt design, it always took me down the happy path. Most AI is designed to satisfy the prompt without really questioning the quality of what it delivers. And then the bad surprises start showing up. A commercial application needs much more than a prompt can deliver: security, usability, deployment, maintenance, users management, cloud, web or native design, and of course quality. It takes a full stack to build one. After months of trial and error, I reached this verdict: -You need a watcher to keep the AI from hallucinating and drifting. And facilitating the route if issues appears during developments and tests. -Never let the developer agent test its own work and claim it's done. -Build a reviewer from a different company's AI to refute the developer's claims and build its own tests. -Let the user be manage the final delivery gate I also built the plumbing to keep the context small and make it easy for non-coders. You describe what you want and approve the requirements. Then the agents build and test it on the machine the software is meant for, with no one needed until you review the delivered solution and approve it. It builds web or native software for Linux, Windows and Mac. It's open source: https://github.com/nomorehappypath/nomorehappypath

Thank you, this is clear and helpful.

Hi, very interesting. I am curious: when an agent says it is done, who checks that the agent at least did due diligence tests, and who checks that the real end-to-end tests were really done, maybe based on a list of scenario situations created by agents to be tested?

I am right now on level 5.

This is a problem I've been working on for the last 6 months. Building commercial apps with AI (agentic development), and not just small, nice-to-show apps.

My problem is that when the agents used to tell me "Done", in fact it was not done. It failed to test real live situations; it tested enough to show me it works, in a nice way, I should say. But in reality, it can never be commercialized. To have a commercial solution, first it has to bring value, then you have all these things around the solution that need to work: usability, help documentation, error catching, monitoring, bug fixing (maintenance), security issues, deployment issues, user rights issues, subscription issues, ... There are a lot more than just the "it does the job" thing. The whole machinery around the job has to work.

So coming back to the question: can an AI agent do that? NO, not alone. It needs a whole machinery around the agents to build a commercial solution.

I am on the final touches of my open source project (yes, after 6 months of hard work ;-)); if interested, I can share what I have done. It already built 2 commercial apps used by real companies (one marketing solution, and one commercial land analysis in Texas).

I agree with you on this one; the compiler output remains the same if run the next day, but AI code might not. That is why you need a strong, independent reviewer who reviews the AI code and freezes it after verification, so that next time you do not have to ask the AI to regenerate the code.

Fair assumption. I think humans, at least those who like to code and build solutions, will become better builders than coders. Understanding the code will become less important, but knowing how to put it all together will be essential.

I agree with you. A code reviewer from the same company, even with different models or efforts, shares the same blind spots. To review my code, I use a different company; if I code with Claude (usually Sonnet 5.5-medium), Codex reviews it, and vice versa. I get much better code this way. But more importantly, I always tell the reviewer to build its own test-scenario ledger and refute the coder's claim that it is done. And the verdict is shared with the coder, but the coder cannot reject it.

very interesting, i will give it a try. Thank you

Very interesting using subscription instead of API.