Scaling Git at OpenAI
openai.com
openai.com
- Cursor: https://cursor.com/blog/git-at-any-scale
- GitHub: https://github.blog/engineering/architecture-optimization/bu...
- Anthropic: https://www.anthropic.com/research/redesigning-source-contro...
- GitHub: https://github.blog/engineering/architecture-optimization/th...
Maybe also tangentially East River Source Control's too: https://ersc.io/blog/what-comes-after-git.
I'm wondering how much cross-pollination of ideas and subsequent implementation details there are between people working at these companies. It does seem like all of this could potentially converge.
Which would support your argument, but I don’t think this is particularly new.
I think their article reads a bit differently, they haven't optimised git, they've ditched it in favour of a new ~~CVS~~ VCS.
I don't think so, their article says they were seeing 30,000 commits a day in September, when I was at Meta six years ago they were easily doing that.
Seventh, at least, all the major code forges are on it: https://about.gitlab.com/blog/scaling-git-for-the-agentic-so...
Entire also seems focused on this: https://entire.com/news/entire-launches-distributed-git-netw...
Supposedly they started gzipping the conversations.
On the other hand, some lower-level tools have to be simple, reliable and efficient. Here, sharing is critical. Even the authors of the post shared some incremental improvements to git. Here, the scalability story is moot.
I think git is becoming a protocol. One product can no longer cover these cases.
"the entire point of git is moot"
The point of my response is that the analogy was broken without saying what adding those pages could provide.
The analogy isn't broken. Your objection didn't really make sense, because adding pages to the tax code makes the tax code strictly more complicated. Your response just ignored the case where that complexity actually matters.
I am watching OpenAI's output, and while a lot of it is absurdly impressive, I can not even begin to imagine what kind of activity of theirs justifies a through-output of 5 petabytes of -GIT COMMITS-. That's not their data sets, that's not their models, that's not their math library. That's their code.
They're rengineering Git for scale, but like, why is there a demand for scale at all? If i saw an employee flipping the same number on an excel file over and over, I'd fire them, and yet they're perfectly happy running agents that do that and setting up meetings and writing blogposts about how much they improved their spreadsheet number randomization factory.
Why would the volume that any of this be increasing exponentially? And what are all the engineers doing when (besides the model quality) their products don't seem to be changing very much?
As a user of Astra, it writes code that appears to work. It does not necessarily write sensible code that is architected in a way that makes sense based on what it does. I just watched it do a major rework of a couple thousand lines of code by keeping the original organization and kludging up an adapter to make old code have new semantics.
I would expect that this approach, coupled with lots of fun asking for changes and none of the grunt work of actually thinking about the result, could result in exponentially increasing amounts of mess.
I haven’t used a recent Anthropic model - maybe it’s better at this.
just exercise. it will be ok, really. somehow we managed for millions of years, kiddos, and both gravity and the human body are unchanged in any relevant way since which suddenly requires "an app". I can get out of bed without an app, or an AI. I can prepare a meal and eat without any kind of app, or any kind of AI or LLM. its a true and real thing, in the real world, promise!
I would have loved to have all that in an app over the last 25-30 years so I could see my progress.
There are tons of different conditions people are in, with different histories of injuries, etc. and even people without any issues benefit massively from having a sensible, well structured workout plan that ensures their weaknesses are addressed and they work towards their personal goals. You don't need AI, an app or a personal trainer for that but while many people do enjoy learning a lot about exercising in their (often limited) free time, many don't and it's great if they have an affordable/free option. Why do you think physiotherapy exists?
Additionally, studies have proven that progressive overload is a really important aspect for progressing, so you should be tracking your workouts in some way, in an app or on paper, to ensure you really are challenging yourself.
I hope this practice doesn't die out, or complexity will skyrocket for no reason.
It writes like it works at big tech and is angling for a promotion to L{n}+1 with this "project".
If you are wondering what these changes are, you can take a peek at https://github.com/openclaw/openclaw/commits/main/
This isn't to say that such pace of iteration is good or bad. That's a separate discussion.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
I think it’s simply a matter of dogfooding and new feature development, they do have much higher velocity than companies of equivalent size and an unlimited token budget helps as well.
I do believe this will force the AI labs to eventually solve the bloat problem with AI, but we’re definitely not there yet.
Give a verifiable benchmark for the AI agent to hit such as reduce token processing time by 5% without affecting output.
Kernels, performance-required stacks etc. is different, so you optimise them for that.
It feels like OpenAI and Anthropic have a ton of room to optimizing that regard, but they choose to rather focus on moving the frontier forward than doing things that are already possible in a more efficient way.
It's better than the ductape and glue pipelines that we had previous to this.
The software is the definition of plumbing no one cares about and testing it is trivial.
many yaks to shave. many bikesheds to paint.
if you're paid by the line, why not go to town...
Can’t roll out fusion powered data centers in 2030 without a simulation engine.
Because the limiting factor for software has always been the supply, not the demand. When the average human engineer would ship 100 lines of production code / week, you could only afford to do the most critical things. But if code gets cheaper to produce, the projects that were marginal become worth doing, and the projects that were obviously insane become marginal. And that's exactly what is happening as models get cheaper and better, leading to the frontier of worthwhile projects expanding rapidly.
> And what are all the engineers doing when (besides the model quality) their products don't seem to be changing very much?
That's got some nice But other than that, Mrs. Lincoln, how was the play? energy to it!
So you think that the move from 4o to Astra required something like 1000x the code to pull off? I find it hard to believe as an MLE, but since I don't work with LLMs I might be wrong about how complicated the implementations have become.
I do know people who work at OAI and it seems like none of them work directly on the models, but that's n=1.
Surely it's obvious?
Slop cannons firing at a rate so high no human can hope to restrain them.
Such a colossal volume of slop that even the most well engineered, efficient technology on the planet, written by one of the greatest coders of all time, is no longer sufficient to contain it.
And git is enough to version control some of the most complex software like the Linux Kernel, the FreeBSD operating system, QEMU, browsers like Firefox,... Git was insufficient for Facebook and Google because they wanted to put the whole org inside a single repo, which broke the assumption where decentralization in git would happen (repo level, not subtree level). Now it seems that another assumption is broken, that you wanted a version control system, not a snapshot or a rolling log.
In doing so, they obviously need to keep track of the myriad of changes, hence the requirement to ensure Git can scale up.
(I'd like to think /s, but you know what, maybe not)
Of course, then the question is if it's even worth committing them - you might as well just commit the design spec and let the model generate ephemeral code to run this whenever you need it. For now, I almost consider committing generated code a caching step in some sense, so you don't have to spend the tokens and time to generate it again.
Once super fast and super cheap tokens become more ubiquitous, however, I expect this to change.
What about user testing though?
Or it's yet another iteration to get another test with overbuilt providence catalogling, boated code, overbuilt nonsense. Suddenly you're 2.5 Billion tokens deep and you just wonder "What the actual Feck did they do, how did they burn that many tokens? It's just a web app with a plugin system, my god."
[0] https://martinfowler.com/bliki/ArchitectureDecisionRecord.ht...
I've seen it on other workplaces, but from what I see from open ai and anthropic, the code shouldn't be as extended.
But maybe I'm crazy lol
Disclaimer: I am building a code forge at https://juju.bi
Impressive!?
Yes! Great adaptation of an existing system, showing why open source is special.
Neat hack, OpenAI engineers!
Monorepo. I keep hearing from various folks across the more frontier tech companies, that they have switched to a monorepo to improve agent-scale dev performance. Anyone have similar findings?
The same is really true for agents. All code in one place makes it faster and more reliable to load the proper context for coding tasks.
If they switched to private packages in separate repos, would that scale better? Would they be hitting the same problems? Would they be hitting different problems?
This thing aside... IDK, I used to think that there's no way to manage such a mess but apparently there is. Or rather, it starts to seem plausible that models will evolve faster than this mess, and once it becomes unmanageable, you can simply ask AI "make it better, no bugs please" and it'll mostly work.
It's becoming harder and harder to justify my own existence as a software engineer. It's surreal that that I dedicated years of study to prepare myself for a job that's being automated away.
Of course the demand for human developers won't drop to 0 but it'll be limited to a handful of industries where even small mistakes are costly, like aerospace. But then let's be real - what percentage of currently employed devs are competent enough to write such code?
That coding volume is a liability only for human engineers, with their limited memory, short attention span, tiredness, and quirky coding preferences. LLMs don't have any of those problems.
LLM compute is getting cheaper, so one can re-run the coding agent until it produces the desired result.
> without adding value compared to a more thoughtful, efficient implementation
Thoughtful implementations require thoughtful developers, and thoughtful developers cost more money that dumb GPUs.
Thoughtful developers do take money, but so too does shitty code that reaches a point LLMs struggle to maintain. Having both- with the thoughtful developers in charge- is better than just throwing more LLMs at it and hoping for the best.
If you seriously believe we have "solved coding" or achieved AGI in coding, what is not possible? Any argument you can come up with will be easily dismissed with "we'll just ask the AGI to solve that problem".
Perhaps I'm being overly generous to the parent's point, but it stands to reason that we make a lot of choices of where and how much trust we give away. This is just another one of those points.
Trusting that a calculator works is like trusting that there will be another day tomorrow, while trusting the code of an LLM is like trusting that the current weather will hold for the next hour.
Also, LLM finding bugs in human written code doesn't mean it wouldn't find much more in LLM generated code, and spend much more money doing so. That is, of course, after ruling out the false positives (of which there are a few, judging by the PRs I've received on my open source projects)
Frankly, these discussions are tiring because there's a false notion that humans have to be in the loop 100% of the time. This is untrue. LLMs are accurate enough that they can be left to do things on their own and human intervention is needed only for out of distribution instances.
The manager always has a very good stick: “Do a good job or you will be fired”. And me wanting food is well aligned to not produce buggy code. Or make my job harder in the future by writing unmaintainable code.
> Frankly, these discussions are tiring because there's a false notion that humans have to be in the loop 100% of the time
The code is written to process data. Someone needs to be aware of how it’s processed, so that when malformed data appears, we can collectively find where it happens. You can always play around with your own data (if regulation allows) but you need to respect your customers’ data. Not looking at the code jis just negligence.
You are welcome to not use the model if it makes mistakes. Its weird that you are cooking up some edge case scenario to make your argument strong - everyone is incentivised to make things work well. Anthropic wants their models to perform well because that's why you will buy it.
> Someone needs to be aware of how it’s processed, so that when malformed data appears
Eh no. You need to know about only a compressed part of it, not 100%. Do you really claim that you need to know 100% of the code? This is not true today and certainly won't be true in the future. Hardly any code will be looked at by humans in the same way my manager doesn't look at my code.
What Anthropic wants and what happens in reality are two different things. But with code I write, I do control the process.
> Eh no. You need to know about only a compressed part of it, not 100%. Do you really claim that you need to know 100% of the code?
I know the code I write, and I know how it interfaces with external modules. When those interfaces don’t function as described, I look at the code if I have access to it. When I don’t, the party in control has always good incentives to produce good documentation. That knowledge is usually good for some time (years even)
> my manager doesn't look at my code
Because that’s not his responsibility, just like he’s not responsible for bugs and crashes. He’s responsible for the project and the team and you can be pretty sure that he’s looking at all at the information about those. Imagine if his own manager ask him about some details and he answers: “I don’t know, Claude is doing it”.
This kind of fallacy keeps repeating time and time again. Another version of this fallacy goes like this: "agents can create a lot of code but humans have to review each PR anyway so reviewing is a massive bottleneck". This doesn't make sense because reviewing is also done by agents and humans don't have to read every line.
Who is this code helping?
Centralization is exactly what you want at this kind of scale. The most marginal participation from the server would make a huge difference.
There is a reason game studios don't use git. It's not just about the large objects.
Wasn't this flagged as a big risk, as it seems to place complete trust in the client?
Especially with all the rogue agents and containment/monitoring failures we have seen in 2026?
Would love to hear how the validation is deemed sufficient.
In all seriousness though I have enjoyed reading these posts. I will never work at this scale, but I like to read how different teams approach problems like this and I do wonder if some of these approaches will trickle down (or up?) into regular practice as everyone is committing more and faster.
Also, maybe FUD, but is there anything useful to learn from (p)npm other than cautionary tales?
This is incorrect.
It actually will work if you mount a drive. Github mirrors can be on separate drives. Now I haven't fully tested it's safety in that situation yet, like I have with local dev on Windows and MacOS.
> Also, maybe FUD, but is there anything useful to learn from (p)npm other than cautionary tales?
This is also incorrect.
Denigrating pnpm's well regarded optimizations because NPM packages have vulnerabilities unrelated to NPM is missing the point.
Over the last few months at Xavi Group, our team of one engineer has gone from ~1k commits a day to ~100k commits per day as a result of debilitating financial decisions and exponentially increasing agentic workflow capability. Our internal Gitea services began to get bogged down by our proprietary frontier coding agent (internal codename `xavi-polaris`) refactoring the monorepos comprising our three flagship projects, one of which of course is our widely unknown RSI harness project.
(At this time we are not accepting any additional requests for our closed private beta, unless you ask reaaaalllyyy nicely $$$)
These demands were increasingly straining our S3 bucket capacity (petabyte scale) and our CI/CD pipelines (3 DO droplets, a macbook mini, and a legacy windows server possibly compromised). Most critically it was straining the agentic wellfare of Xavi sessions themselves, which began to flag push/pull latencies that were taking so long their caches would go cold between tool calls, resulting in even greater financial hardship.
`xavi-polaris` sess-3LQGgyUxLGXiV7OTTIMo4TWm0a6 is credited with identifying that this unprecedented pace was unsustainable, with projections that by mid 2027, commits per day could be north of 1 billion as monorepo sizes increased and refactoring became increasingly challenging despite the profound software engineering capabilities unlocked by the RSI harness project.
To identify solutions, we first looked to ensure we wouldn’t reinvent the wheel we were reinventing. We noted projects at Google, Anthropic, and OpenAI that recently aimed to address the very same challenges, but after ~20k USD in session research that incidentally solved an open conjecture in statistical analysis, we decided that adopting any of these existing protocols would limit Xavi sessions in our internal promotion benchmarks that enable “contributor” sessions to progress to “lead” sessions, irrevocably inhibiting our expansion of upper agentic management.
We’re proud to introduce our new, closed source Git model: AGIt.
AGIt is a drop in replacement for Git, which we identified as being critically important because xavi sessions are deeply familiar with git commands as a result of a massive distillation and theft of proprietary data. AGIt utilizes novel CRDT and distributed systems technology built from our extensive experience with scaling distributed systems off the back of YJS. If that doesn’t make any sense to you, it didn’t make any sense to the individual human engineer at Xavi Group either.
Importantly, we don’t believe it needs to. The future is agentic workflows, and we’re excited to continue giving back to the nursery that birthed our projects and has delivered at scale for almost every other software company on the planet. We’ll be donating five `xavi-polaris` teams to pumping PRs as fast as possible to these open source projects. As we enter this new era we’d like to thank contributing sessions sess-3LQKRQrJUyAvYHlUDJcjos2CInf and sess-3LQKRnqna2emLvyUyPUqSEIaUh0, their teams were instrumental in this endeavor, and both will be promoted from “lead” to “steward” to reflect their tenacity in building this system over the last few hours.
Unlike Anthropic or OpenAI where no single engineer has any idea what they're doing and still uses pathetic source tracking, we have implemented an evolution and moved on to the next stage with our AGI: Wish Driven Development.
Our engineers, now called Wish Whisperers, open pull requests to add wishes to a controlled environment where all our wishes are stored in a Wish Registry. Swarms of agents are then dispatched to validate these wishes and when the pull requests are merged, implement these wishes directly in production.
Our productivity has significantly gone up, so much so that our Wish Whisperers are now running their own Wish Registries to automate their wishing speed.
To be frank, we still have no idea what we are doing but this is AGI level productivity and it's all that matters at the end of the day.