GitHub Incident with Actions, Pages, and Webhooks
githubstatus.com
githubstatus.com
https://news.ycombinator.com/from?site=githubstatus.com
It's driven by aggregated user complaints, so it's pretty high-quality. Comment counts are a proxy for "how bad was the outage" + "how long was the outage thread topical for".
edit to add: Here's a scatterplot of the same,
Disclaimer: I am building a code forge myself at https://juju.bi
Isn't everyone else doing that too?
* https://cursor.com/docs/origin
* https://tangled.org/blog/forge/
* https://entire.com/blog/the-git-forge-for-agent-human-collab...
The space is getting a little cramped, to say the least.
* https://juju.bi - Local-first architecture
* https://cursor.com/docs/origin - Scalable VCS backend (this is actually very easy to implement by others)
* https://tangled.org, https://radicle.dev - Distributed forges
* https://entire.com - Very agent-centric
On top of that the tooling is just... not good. The more I use it, the more I get annoyed at weird UX decisions, strange missing features, and the whole thing just being a giant bowl of spaghetti in general. You can easily spend days trying to make your workflows not suck just to realize halfway through that you could've been done already if you had just handwritten the damn thing in Bash.
We now live in the insane world where I am glad our tickets are still in the nightmare called Jira, because every day I get closer to asking management to switch to a forge which is actually functional - and with only CI and Git the transition would not be very difficult...
I had a batch process I became responsible for that ran on the same VM as an important production service and every once in a while it became a noisy neighbor. It ended up getting moved onto a much smaller cluster for the compute part of the operation after I did some pretty aggressive traffic shaping work to get the cost down by more than a factor of 10.
At one point I got the code into a shape where I thought I could just run it directly on the CI machine, and the lower CPU availability would be offset by the higher IPC overhead and cache misses. Problem was that it was still about 2.3x slower running on 1/3 as many cores. But those VMs idled for hours at a time while waiting for a 10 minute job to be started.
Near as I could figure, if we doubled the size of the instances the build agents ran on, I could shut down that cluster and just about offset the cost of the more expensive build agents exactly, and speed up quite a lot of builds in the process (many build steps do not parallelize but some do).
If one other team had a batch process they could just run directly on the CI machines, that would be a net reduction in operating costs. We had quite a few teams doing batch processing. CI servers have a lot of tricks you can use to queue up tasks rather than running them concurrently.
And yes, I need to optimize (and am), but they're already getting hundreds from little me. I even started going towards the CI doing browser walks, code tests, AI rework, etc. It saved me from having another dev machine.
Great stuff, powerful as all heck, and once you see the power it was—for me—impossible NOT to think of 6 other uses and blow through the free CI tier.
Especially if you can take advantage of the high vcpu runners, then you can get some really fast pipelines going (not affiliated w/ them, just an enjoyer).
[1] It can also have supply chain security and reliability benefits too. You're not actually building deployment artifacts in your actions runners, right? Right? RIGHT?
Github has a decade of positive reputation to deal out, I don't see Grok or OpenAI jumping in and competing with that.
Easy enough to run from another CI or automated system.
But, the wrangler CLI isn't that much of a step up. Biggest items are installing the CLI and having the CloudFlare API token available in a secure manner.
Otherwise I reckon they’d have cloned GitHub already; possibly with a “Mirror to CodexHub? Free usage reset”.
Without free CI in GitHub, you might as well say:
"they can just shut the whole thing down and this reliability issue will be solved immediately"
I would not be surprised if a serious amount of this usage is not user-intended and instead the LLM just inserts some sort of CI job off the bat and people roll with it.
After many outages we had to build our own free runner https://github.com/supercorp-ai/superci
The federated code forge (https://news.ycombinator.com/item?id=50305573)
After everything announced in GitHub Universe and with that to improve the reliability of GitHub overall, yet we have another incident afterwards.
It doesn't matter what GitHub does, only you can decide for yourself to self-host rather than going all in on GitHub and it continues to go down.
Predicted this 6 years ago and self hosting has aged perfectly well. [1]
Takes a few minutes a month to update and babysit. We only basically use the CI and the repository functionality though so gitlab has been "feature complete" for us for a while. Probably I we could swap to something like gitea or forgeo but the gitlab CI seems like a nice balance between powerful when it needs to be and simple otherwise for our use case so there is nothing driving us to swap.