Incident with Pull Requests, Git Operations and Issues
githubstatus.com
githubstatus.com
A sick person with symptoms is clearly sick, but sometimes they're asymptomatic. That's what the GitHub status page has become. When it's red and showing problems, you know you have problems. When it's all green? You know you MIGHT have problems.
That can make outages like this harder to triage, since reality is a black box.
ECR just seems to work really well for most people and IAM is nice compared to Github Apps + rate limits
I assume these companies have some kind of uptime guarantee in their contracts, because they seems to care more about covering their asses than about informing customers.
We have outages on our books that they never reported even after we ticketed them.
They cook the stats like an inner city police department.
Github's status page is incredibly slow in comparison to the status page Github themselves used to have, which had several key metrics of various systems that would update in real time, and when they did have a (comparatively rare) outage, would usually have at least one metric going "yeah, something is wrong".
Don't know about you but at this point I've learned to recognize and ignore GitHub issues even when they haven't admitted to it yet
Question 1: Is it github, or is it everyone:
Answer: https://downdetector.com/
(If everyone's having issues, then stop. Otherwise, continue below...)
---
Question 2: Is it github?
Answer 2: https://www.githubstatus.com/
Question 3: Why are you lying to me, is it github or not?
Answer 3: https://mrshu.github.io/github-statuses/
(If github is not having issues, then check your local environment. Otherwise, continue...)
---
Question 4: Why is it always github, surely they're not that bad right?
Answer: https://www.dayswithoutgithubincident.com/
(Perhaps its time to consider an alternative to github, or at least a mirror)
> November 4, 2026: We’ll have a brownout of the removal of the ssh-rsa signature type (i.e., RSA keys using SHA-1) and the diffie-hellman-group-exchange-sha256 key exchange algorithm.
I guess something went sideways.
Disclaimer: I am building a code forge myself
- - > i wake up - -
| |
| V
github has an outage def is_github_down(day):
return day[-1] == "y"Gitea Enterprise would be worth looking at.
I have it for all my stuff (dozens of repos and hundreds of PRs/builds per week) and it's only once in the last year that I've exceeded the free resource levels. I don't know if it's just extra capacity or they're burning VC dollars in the hopes of inspiring posts like this, but either way I feel like I'm taking advantage of them at this point.
If you're looking for a publicly-accessible Forge SaaS for your private repos and are willing to pay for it, you are almost certainly a company developing proprietary software.
Begin by refactoring the deployments so they're driven by checked in shellscripts, that accepts a git reference to deploy (prefer immutable commit SHAs over mutable branches/tags). Make it use the stuff it already has checked in, and make the actual GitHub Actions stuff as small and concise as possible, ideally each "run" should just call a shellscript with the git reference.
Then move so the deploys happen not on GitHub Actions itself, but on a "intermediate deploy host" that has the source code and access to the credentials you need, and make this host have one secure entrypoint, where someone (or something) can enter and run shellscripts, after pushing their code as a normal git push. Then switch so the deploys from GitHub Actions instead use this new host to do the deploys. Lock down this host hard.
When GitHub and/or GitHub Actions go down, the developer responsible for deploying stuff, should be able to get temporary git and ssh access to this host, can push any git data straight to the host, and their user on this host can run the deploy scripts. Keep in mind that nothing in your deployment process should depend on stuff that isn't also available on this "deploy host", if you have multiple synced repositories and so on.
Now things mostly work the same as people are used to, with the added benefit that when things go down again (like next week), you'll still be able to continue to deploy stuff.
I've done this refactor/migration for tens of companies in the last year, more or less following this exact template for all of them, with no issues reported from either of them, but lots of praise :)
I am requesting to merge my changes into the main branch.
Similarly: Why are the Gitlab people so happy to merge? Where is my "Rebase Request", "Squash Request", or "Cherry-pick Request"?
related tidbit: in git, "pull" is basically shorthand for "fetch + merge".
Once all is good, push into remote HEAD and GithHub should auto-close.
Though, larger project tend to leave the final upstream merge to GitHub itself. Better access control and it’s easier to serialize merges through a single queue for very active projects.
You're literally replacing a well designed resilient system with a single point of failure run by a different company...
Wait, Jenkins?
It's a great service though
git -c credential.helper= -c 'credential.helper=!gh auth git-credential' pull https://github.com/myorg/my-repo.git
git config --global url."https://github.com/".insteadOf git@github.com:
so it pushes / pulls using HTTPS
when things are back to normal run:
git config --global --unset url."https://github.com/".insteadOf
Wait…
Until next time. When I look into alternatives again. Probably, one day, I'll actually migrate away. But for now I shrug and tell myself that these things happen.
Developers are expensive. If I can't ship a feature on time because GitHub issues keeps forcing me to switch to different tasks to work around git/PR/ticket/Actions downtime, then that costs my company real money - despite us not being the source of the AI DDoS that GitHub has happily been welcoming.
To me, the situation is very simple: either 1) rein in the AI deluge, or 2) isolate paying customers from it, or we'll move to an alternative.
For businesses the current state of unreliability would be unacceptable if it was completely free, the fact that it is actually paid just adds insult to injury.
new incident with actions now - https://www.githubstatus.com/incidents/h02n9572mx9y.
What were the 60'000 people exiting Microsoft doing if not improving Github?
I can live with the occasional outage but this is a constant problem.
I joke, the QoL is really nice
> git@github.com: Permission denied (publickey). fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
https://rubyonrails.org/2026/11/4/Rails-8.0.5.2-and-8.1.4.1-...
https://www.githubstatus.com/incidents/xr3myzj7ms51
https://www.githubstatus.com/incidents/h02n9572mx9y
Get your act together, clowns.
Perhaps these types of posts should just be pinned at this point.
Maybe they need a better AI model for their vibe coding. That'll fix everything, right? At least that's what my friends keep telling me when I say that the free LLMs I played around with a bit are incredibly dumb.
It's up to you if you want more of this instability. Perhaps you should consider a self-hosted alternative.
Some of our customers unexpectedly started using the service we're running as a file-cache around GitHub to mitigate GitHub outages.
We wrote about this use case in our blog: https://www.stablebuild.com/blog/protect-your-builds-from-gi...