Web search, steaming, high frequency trading, and many other systems are resource demanding, and keep scaling. Whether to handle the surge due to bots or wider adoption. But GitHub can't scale git? It isn't even git failing.
Incompetence in leadership is what makes a tech business technically unable to meet growing demand.
Is there a single Google service that has ever had reliability this bad? Facebook? Instagram? TikTok? Apple? Those companies all run hugely scaled write-heavy platforms. Why is GitHub so uniquely unreliable?
Me and 1 billion of my best friends can upload 5GB 4K videos to iCloud Photos and live stream to Facebook all day long but GitHub who is owned by Microsoft the second largest cloud computing provider on the planet can’t handle 180 million users pushing code and running pipelines on VMs?
I'll give you an example: I pointed Codex at one of my repositories asking it to implement some features. Very quickly, it ballooned a 3 minute CI workflow to 30 minutes per commit, and the number of commits it started making increased 10 fold. So we're looking at a 100x increase in usage just from myself.
You factor that into the fact that all public repo get access to this same compute, basically unlimited minutes, up to 20 parallel tasks and 6 hours per request. Now because of AI, every small little weekend throw away side project is running a full dev ops shop. And the pressure never goes down because the AI just start a cron job to run these systems every single day, whether anyone is even looking at these repos including the authors, who may themselves just be bots.
And the systems are sophisticated: parallel builds across windows, mac, multiple linux distributions, multiple architectures, embedded targets, web targets, full feature matrix, the works. The default MO of these agents is to do as much testing as possible without any regard for resource constraints, and GitHub offers them unlimited resources, so it's like an addict meeting a dealer.
So in a sense, maybe being attached to Microsoft is the problem. Or at least their unlimited wallet is -- Microsoft is the enabler in all of this.
It still just sounds like a really vanilla "horizontally scaling VMs" problem, especially for yesterday's incident that was focused on pipelines.
If this is a "GitHub is giving away more capacity than it has" problem, that's easily solved by rate limiting and queuing.
This is GitHub's explanation:
> During the incident, some Actions Runner Controller (ARC) runner pods became stuck in an idle state. Affected users can delete those pods using kubectl or redeploy their Actions Runner Controller application. ARC will automatically create replacement runners.
> The next releases of Actions Runner and Actions Runner Controller will include an automatic recovery mechanism, preventing the need for these manual steps in the future.
That sounds a lot more like a major architectural flaw and a really obvious oversight.
I don't think this is true. Take LLMs for example, you need GPUs to serve them and these are in short supply at the moment making it hard to meet demand. I don't think this is necessarily the fault of incompetent leadership.