One reviewer per layer: splitting code review by the way code is read | Hacker News Reader

One reviewer per layer: splitting code review by the way code is read

djordjepuzic.com

31 pointsdjordjep12 comments

I think it is a good approach for this year, but it misses something important - how do you know the code still reflects your intention after steering for weeks and months, across many sessions? Messages flow in the session and zoom out of view, and across compactions/new sessions they get forgotten. To solve this, I log every user message in a chat log, and review it from time to time to distill the latest form of user intent.

Another kind of review not mentioned here or in Anthropic's article is the plan review. I generate the plan for some stretch of work, then have the plan (saved as a file) reviewed by one or more agents. This checks the plan against project constraints before executing it.

If we can solve testing then coding is almost fully automated. The only remaining problem was always trust, why should we trust that code? If you read it manually then it's like walking a motorcycle. If you don't, then you need some way to derisk it. There are many ideas flowing around about how to do this, the OP and Anthropic showed their takes. I think the approach is project specific though, no general method.

What I do is that each of my requirements are captured in a yaml file, with a unique id, with the exact sentence I wrote. Each must be covered/proven at least with one test, a linter checks whether the requirement is covered (test exists annotated with that id), and a test inventory generated from it. Usually I browse and review the test inventory first to see I get what I asked. It does not guarantee that the agent implemented everything correctly, but if I ask another agent to review it and point out the gaps, it produces very good results.

Is each file just one sentence? Trying to get a better sense of how you implemented this.

You can see it here: https://github.com/gofabrik/oauth21/blob/main/conformance/ca...

I am experimenting with the process with building up an oauth 2.1 server based on the RFCs to see whether it is a viable way to develop software or not.

Reviewing code by layer is easy if the code is written layer by layer. When I used to code, this ways the highest level of craftsmanship I always aimed at, and rarely hit. It required excellent understanding of the problem, a good plan, and disciplined execution. This was rarely possible in a ship-ship-ship business and a family life.

My ideal state was when I had a PR and each commit had a different layer - DB, controllers, feature A biz logic, feature B biz logic, APIs, backend UI prep, frontend UI, polishing, bug fixing, test cases, touchups, etc. It felt marvelous when it happened.

With AI, this is doable much more easily. Even if an LLM produces one giant commit, just ask it to refactor the code and recommit it to a new branch with one commit per layer. This is the kind of "translation" work that LLMs are now excellent at. Reviewing such PRs is a pleasure. The work to build the code layer by layer is one prompt.

My problem is right now that every kind of reviewing process adds un-needed complexity which later slows down your agility and increases your cost and performance degrades.

If you have feature and 6 different reviewers go over it the logic bulks up easily. I wonder how other guys handle that? I’m all up and I want this to work, but this is my current constraint.

I often get bad suggestions from reviewing monitors because they get something wrong. Sometimes I get good ones, very good ones, but some are bad. So I’m always down to look at reviews about bugs, but I would be afraid after 6 reviewers its just a bunch of text and its confusing me so end result is worse as me looking through actual critical stuff. My context is limited too and completely hands-off is not reliable enough...

How do you guys do it?

I wonder if we would see more engineering teams transitioning away from git to Sapling. We do need a refresh on how we review and ship code.

I'm aware that GH recently released stacked PRs, but it still seems like a long way to go both from usability and reliability perspective compared to what Meta and likely Anthropic have internally.

I am playing with similar ideas.

I am inspired by this to make a harness that intentionally does sparse checkouts, so the agents only can see so much context.

I am also playing with the idea of layering additional data structures on top of the "code" that cover different aspects. For example, a tree structure of performance requirements that has durable symbolic references to entities in the code that are especially relevant. Another tree can cover aspects of security, another legal compliance, another UI guidelines.

Benchmarks or this is no better than my one line "do a code review on this change"

github stacks baaabyyyy

I wrote this after reading how the Claude Code team reviews code. Their layers are stages in time. I propose layering the reader instead: one agent per layer of code (form, local meaning, wiring, runtime, use), crossed with lenses like security, history and reversibility. Each cell gets its own narrow context. Curious where people think it breaks.

this is common practice, works pretty well in my experience both to improve quality and reduce cost, and afaik is a naive version of what tools like coderabbit and greptile do internally

Reply on news.ycombinator.com