vivekpolavarapu

Taking 5x longer to review is something we experienced too. While there are multiple code review tools, I am not sure they come close to level 4 autonomy. We have been experimenting by using multiple models to review not just the code, but design, architecture and the agents' riskiest code commits. We just launched it for others to try it out too - https://truverif.ai/panel-review

The main insight was that a multiple models arguing over a code change produces far better insights than a single model review.

Interesting! Will try it out. The semantic diff makes it easier for the reviewer to understand what is being shipped. This could work well with the Panel Review we shipped recently -- It reviews the agent's riskiest changes with an AI panel. This reduces blindspots and minimizes agent mistakes. More here: https://truverif.ai/panel-review

Given that each model has its own blind spots and jagged intelligence, I used to run one model's output by another to get the most reliable outcome. Then it occurred to me that humans always arrive at truth through argument and consensus (e.g., courts, sciences etc.) I applied the same principle to built https://truverif.ai/home that utilizes multiple models (arguing with each other) to arrive at the final outcome.

While looking for practical applications, I realized coding is the space that needs this most because it has the highest AI agent adoption and the mistakes made by agents cost 30x more if they are discovered in production (added to the loss of customer trust). So, we developed Panel Review - it routes an AI coding agent's riskiest diffs, designs, and commits through a panel of four frontier models (e.g., OpenAI, Anthropic, Google, and xAI). Tool has review gates the agent cannot skip. We built these gates after learning the hard way that agents don't follow written instruction to do a review. In ten weeks of live production use, the panel found at least one critical or major issue in 46% of the reviews and 70% of the reviews resulted in the agent changing implementation based on panel recommendations. For example, one review caught a prompt injection hole in our own MCP server. Happy to answer any questions.

Very interesting. Will try it out. You called out the problem at the top of my mind - AI agents do well sometimes, but "will fail badly on some problems". That said, we took a different approach to address this - Panel Review - https://truverif.ai/panel-review. It identifies the agent's riskiest decisions and runs them by a panel of AI models (user configured) that argue and provide severity-tagged findings. The panel ensures that a single model's blind spots don't become the vulnerability.

Interesting. While I am unclear if Jev can indeed be a standalone reviewer because it cannot generate conversational text or free-form comments. Maybe something like this - https://github.com/devagrawal09/jev-review

This sparked an idea for me to use Jev in my recently launched Panel Review tool that reviews the agent's riskiest decisions using a multiple models that argue and come to a consensus. see more - https://truverif.ai/panel-review. In order to determine risk, we use a text based classifier which I think could be replaced by Jev. Will still have to evaluate the privacy and cost considerations.

Ad-tech spent 20 years trying to infer intent from clickstreams. Chat apps now hand over an AI-written one-line summary of intent, labeled and keyed to a cookie. Are we sure the ads business model in AI is about ads in the chat, and not the chat as the targeting signal?