rafiss

The Migration Assistant was mostly a greenfield project and the roadmap was steered directly by feedback we continually received during development. It was built on top of our existing CLI-based tools for migrations, but the web UI and step-by-step wizard with AI assistance was all brand new.

What worked was a human writing the big-picture issue and the hospital decomposing it into a tree of smaller issues, which people could review with more rigor. We also did a few thing that didn't work; in one case we got a bit too over eager in trying to resolve a large slate of issues all at once, and left ourselves with a giant backlog of plans and code to review.

We're discussing internally what can be released in the open, but nothing to share yet.

The closest thing we have is a fuzz testing framework that generates random schemas, drives the Migration Assistant end-to-end using Playwright, has an LLM judge the results, and files issues back into the hospital. It’s run manually, not after every merge.

Funny, right now we have a super bare bones of something like that. When the Chief (a human) makes a decision that generalizes (for example, a project-wide best practice), it goes into a precedent log. Before an agent escalates something to the Chief, it has to check the log, and if a precedent applies it cites it and follows it instead of escalating. It’s been one of the ways to cut down escalation noise but it definitely could be more mature.

Fair, the codebase might be more accurately described as the patient and the bug is the ailment it's being treated for.

The original idea of the metaphor is that a "github issue" is a patient that enters the hospital, gets "treated" (by being resolved), then leaves the hospital. The metaphor gets mixed up when we start using the word "bug" and "github issue" interchangeably.

Thanks. Yeah I agree that it’s not always a good fit. There are plenty of places where we bent the metaphor or just ignored it; like for example I also added a "test classifier" checklist that is really just about software engineering practices with no attempt at connecting it to doctors.

As we consider bringing this pipeline into larger repositories with more developers, we have discussed whether it's time to drop the metaphor. The last thing we want is for someone new to a project to have to learn the inner mechanics of the medical industry at the same as figuring what the hell is happening in a gigantic codebase.

We have a small version of this, but as rules rather than an agent. A token guy role that looks at tool-call patterns and cache use could catch a lot of the wastage that we're missing.

Yes, the codebase still allows you to make a PR the normal way, and getting another engineer to review. That is, a human could make a PR totally by hand or by steering a Claude session on their machine.

I still find myself taking that approach fairly regularly. For most framework-level changes, or changes to core packages, I think the added scrutiny and deliberation is valuable. But for "leaf" components that don't have many dependents, using the hospital to move rapidly has been invaluable. I also encourage "old-fashioned" PRs for anyone who wants to gain a deeper understanding in a certain area of the code.

I like the control-group idea. It’s close to something we’re already building, where a human can label an issue to opt-in to write the plan or the code themselves. Routing a fixed % of issues that way would give us a baseline we don’t have right now.

Rather than per-day costs, I think it might be more meaningful to look at per-result costs. It took us about 9 months (and $160k using our napkin math) to add support for Oracle. Adding Db2 support using this agent pipeline took 2 days and $4,172 in tokens.

One caveat to the number for DB2 is that it doesn't consider review costs. We weren’t reviewing the hospital’s code during that sprint. Since we’re still confirming correctness for Db2, a more accurate comparison would add the human time to verify this.

I’m not going to claim this tripled our revenue. But revenue is also a function of time to deliver too. When a missing feature in the migration tool is what’s blocking a customer’s migration or a deal, shipping it in days instead of months can move revenue directly if it means the customer doesn't choose a different alternative.

The closest thing we have is if a merged change has to be reverted, the Safety Department holds a Morbidity & Mortality conference and writes up what went wrong. A patient that needs three or more rounds of rework gets one too. There have been seven reverts so far, so not many funerals.

Sorry about that. The repo is internal to Cockroach Labs for now. It was an issue about adding a progress rollup comment to decomposed parent issues. We’ll get the post updated.

I like this. Today the parent issue closes automatically once every child is checked off, and nothing re-reads the parent’s original ask against what actually shipped. There is a deferred-scope auditor that catches the deferrals an agent admits to. Your "verify node" would catch the ones it doesn’t, which is the more dangerous kind. I think it would fit naturally as one final child per decomposed parent.

We don’t run the most expensive model everywhere anymore. We started on Opus for every stage, then reshuffled several times. We landed on Fable doing planning and code review, because code review is the last gate and its bug findings were worth the most. Opus does coding and plan review, and Sonnet runs the nurse roles. The planner can also route mechanical changes to Sonnet. The plan reviewer re-checks that choice, and any rework goes back to Opus.

That’s tuning from the cost data, so it's not just totally based on feeling, but it's not a controlled A/B test. I agree that would be worth doing, and I think our experiment here is helping us make the case that it's worth spending more time and effort on running more controlled evaluations.

I think that’s the real tradeoff. The metaphor brings in behavior we never wrote down, which is both a part of why it works as well as a risk. One reviewer agent declined to send a PR back over a docstring nit “while the patient is exhausted.” It was a fair call, but there's no explicit instructions in any of the prompts/skills that said to do that.

We contain it by keeping the persona to a few sentences and writing the rest as explicit scripts or workflows. The rules that really have to hold aren’t left to the role. Human merge approval is a script checking for a real GitHub approval on the PR. A CI failure or a merge conflict bounces the PR before a reviewer agent is even launched. So the metaphor shapes judgment calls, and the hard rules about what should or shouldn't happen are in code.

I think this is definitely a legit failure mode with this model. We’ve hit some parts of it. For example, a plan reviewer kept pushing a technically valid but negligible security finding through five review rounds and about $70 before a human closed it as won’t-fix. The "Research Department" ran every three hours and was filing around 8 new issues a day, more than we could review. Review nits used to spawn new issues of their own.

What helped us maintain a degree of sanity is that every stage records tokens, time, and cost in a ledger, so these show up as numbers rather than vibes. The fixes were mostly about giving a stage permission to stop. Plan review can now propose won’t-fix when the exposure is negligible. Research runs daily, with a cap on unreviewed proposals. Small fixes now ride along in the PR instead of becoming new issues.

The other thing is that the quality bar should match the domain. Our migration tools move customer data into a database, so our question was “would we be embarrassed to have merged this?” For small apps I’d agree most of this is overkill.

To add a bit more: it was motivated by a general desire to not let costs spiral out of control, and when I read the Anthropic context-engineering post I wanted to apply some of the lessons.

The audit was agent-driven, but it worked from a rubric I came up with. For example: - the same rule restated in several places - narration about what other stages do later - history and rationale that don’t tell the agent what to do

One analyzer agent per skill file flagged passages in each category with a word estimate. Each also produced a “keep” list of things that must not change: commands, gates, templates, sentinels. Agents also did the trimming with some mechanical rules: no command, bash block, label, template, or numeric threshold could change, and we checked the diffs for that. The first pass cut about 18%, because it kept anything it wasn’t sure about. A second pass used an auditor plus an adversarial verifier for each file, and took another ~600 lines out of the six worst files.

The more durable result from all of this was a short style guide for editing skills. When an agent makes a mistake, the natural fix was “add a sentence,” and that’s how the files got that way. Also, long inline bash blocks moved into scripts with their own tests, and extracting them turned up a couple of bugs that had been in the skill file's prose.

This is a good idea; so far we've only had a few informal chats around adding something like this. Right now we play the role of Insurance Rep ourselves by manually checking on our cost dashboards and investigating expensive treatments. (As did our Director and VP on the day when we experimented with using `fable-5` agents for everything.)

> The Licensor may make an Additional Use Grant, above, permitting limited production use.

See the text of the Additional Use Grant for CockroachDB [1]:

> Additional Use Grant: You may make use of the Licensed Work, provided that you may not use the Licensed Work for a Database Service.

> A “Database Service” is a commercial offering that allows third parties (other than your employees an contractors) to access the functionality of the Licensed Work by creating tables whose schemas are controlled by such third parties.

(disclaimer: I work at Cockroach Labs)

[1] - https://github.com/cockroachdb/cockroach/blob/master/license...

We do have compatibility gaps -- some big, some small. But I would still call it compatible because it's definitely close enough to use a PostgreSQL driver with it in production. I would be curious to hear your opinion on which incompatibilities are most important to address.

There certainly are pain points. I don't work on this myself, but one of our other engineers wrote this blog post [0] that discusses the experience of running CockroachDB in k8s and why we chose to use it for our hosted cloud product. Another complication mentioned in there is about how to deal with the multi-region case.

[0] https://www.cockroachlabs.com/blog/managed-cockroachdb-on-ku...

Our goal is to make switching over from PostgreSQL easy. For Ecto in particular, we have a tracking issue that shows the items in our backlog that would improve compatibility: https://github.com/cockroachdb/cockroach/issues/33441

Thanks! I do think that a columnar on-disk representation would likely help with large analytical queries. But I would imagine they would negatively affect performance of point transactions in an OLTP workload. Note that this vectorized engine we built will only be used on a given query if our SQL planner estimates that a large number of rows will be read.

github: rafiss

Thanks! This might help a lot for an idea that my friends and I are working on.