Degraded availability for GitHub Actions, GitHub Pages and Webhooks
githubstatus.com
githubstatus.com
Github seems to have degraded measurably after they got acquired. As is tradition.
This old chestnut.
I guess I can embrace the corporate way, repackage the old solution with a few improvements as something brand new, and pitch to execs.
Yeah. Enjoy the rest of your day off. Winter's coming, maybe go hit a park for a few hours or play a new video game or something. Or take some time to learn a new skill or something. Not your fault if the company can't provide the tools to do the job, go find something better to do.
I was working in the Techstop (40) @ Google, 2011-ish, when a nice person from PeopleOps came through and politely requested my admin powers to help them install Steam on their corp-issued Windows laptop.
I'm sorry, what?
Well see, PeopleOps was going to have an onsite intending to help recruiters better understand Engineering types, and part of the whole business was going to involve everyone firing up a copy of Portal and playing a puzzle-solving video game.
So I immediately got to it. Never heard back as to the success of the endeavor, but turns out there are circumstances where you can break out your Steam library at work.
I had a knockdown drag out fight with both HR and Security. I agree that it was probably better for him to have packed his own personal laptop or similar gaming device but security had shown absolutely no risk and it was clear that HR was just 'clucking to cluck'. Eventually they relented with a mealy mouthed 'warning'. The fact that he had installed steam after his stay had been extended and they couldn't show any evidence that he was playing during working hours was a hillarious, and lucky, footnote.
(Man, unbelievable the mileage we're getting out of this comic.)
I mean this really seriously. Maybe not a game on your computer, that's unprofessional (unless your TL/manager does it first). But
> Not your fault if the company can't provide the tools to do the job, go find something better to do.
is just Truth. What should I do in this case to best deliver value to the company, fly to SF and hand Github a ringbound volume of my changeset? Patch our corporate network so we can interface to Github via RFC 1149?
More "professional" ideas:
- read a book; a novel or TAOCP for nerdmaxxing
- play a game on your phone
- learn to draw. Draw hands!
But sometimes it’s a catch 22. I’ve had numerous times where systems simply didn’t work.
I brought it to my managers attention and they didn’t care.
Then later I leave in disgust or get fired. All my fault. I should have used my magical powers to convince my manager he’s wrong while not making him feel stupid.
Ultimately if you’re on the titanic it’s easier to grab a life boat vs trying to avoid the ice berg.
I personally use Forgejo + their built-in Actions stuff (mostly GitHub compatible), have one old Linux desktop as a server which runs Linux + Windows runners, one Mac Mini for macOS/iOS builds, using NixOS so I don't have to remember how things work, and it just keeps running 24/7, always there when I need it.
You can do your own ci/cd runner system, but you still rely on webhooks or triggers, when you aren't hosting the code.
Yeah, that used to be the single favorable argument for GitHub, nowadays I think it might be the opposite, at least personally. If the contributor can't be arsed to join the community proper (and maybe get a sign-in code to their email, or whatever login you use), I think I prefer life without their drive-by PRs.
Sounds like it's not as self-hosted as you think, or you have some other unnecessary dependency on something here? As long as GitHub isn't the sole place for your SCM, why would your self-hosted CI runners go down just because GitHub goes down?
If you're already hosting the runners, why not host git too? It's even simpler than running your own workers/runners.
2. Why aren't the CI runners self-healing?
2. because when github's actions api is down in certain, seemingly very common, ways even self hosted runners can't pick up the jobs
The pull request system and related machinery is useful, but Gitlab and the ilk can do the same. Even self-hosted solutions like Gitea. But I don't particularly have a problem with that living in Github. If there's an outage, break glass, force push to main.
The actual Git repository hosting part is trivial. If that breaks, Git is distributed anyways, push and pull to any random remote storage. All you need is SSH. The main headache is that automations would break as they are usually hard-coded to a given origin. That is solvable in a pinch, and we have setup mirrors in the past.
It really depends on your use-case and how much you are tied to them (Github authentication?)
I imagine very few people use the wiki, because it's so subpar.
The issues are a problem, but this is why I always thought issues should be a core part of the Git repo, checked into it, sort of how Fossil does it. And so should the wiki, for that matter.
I'm not disagreeing with you that it has degraded, however it was acquired almost 8 years ago at this point, and the software world has been turned on it's head in the last... 3?
Software generally degrades post-acquisition, I'm not denying that. Github has definitely degraded. I'm not sure I would agree that the core reason in this case is "the acquisition" and not "AI making software and free-CI-load demand 3-5 orders-of-magnitude higher than it was 3 years ago"
Either way, it's clear the acquisition has not helped github in any meaningful way. GitHub is owned by a multi trillion dollar transnational corporation. They have enough money, resources, and human labor to achieve whatever they want, but this is what they want and this is what they've achieved.
The thought of migrating something at the scale of github to azure gives me the cold sweats.
The problem isn't Azure. Azure is the solution. They have written about this a lot. It's easy to research. You could even just ask ChatGPT to explain it.
Even taking into consideration the fact that Google never wants to talk to you for any reason, it has been my professional experience that Azure is -by far- the worst of the Big Three "cloud" providers.
No
Would data change your mind?
Whats the deal with the colors? Red and yellow don’t strictly seem to mean higher/lower. Some yellow dots are lower than red.
On the other hand, I don't remember things being this bad before the acquisition.
Their explosive growth caused by LLMs seems like it must be a much better indicator than the acquisition. I bet Microsoft are losing a shit ton of money on GitHub right now and seriously questioning the decision to buy them.
You're basically claiming they are lying and the sudden development and execution of an Azure migration plan last year was for other reasons and they just blamed it on AI.
We have a term for that; conspiracy theory.
Edit: here's the link: https://isolveproblems.substack.com/p/how-microsoft-vaporize...
They could, hypothetically, filter bot traffic; I believe they are probably avoiding that solution because they don't want to be, as a market position choice, "anti-AI" (and you could blame the acquisition on this; I'm not sure pre-acquire GH wouldn't have responded to the AI craze with "LOL, no" and instituted a "no agent-driven PRs" policy).
it can
> They could, hypothetically, filter bot traffic;
Nobody seems to be able to do this reliably for a whole huge list of reasons, including: the LLMs themselves can simply mimic human access patterns, timing, and behavior.
I will never trust github at any other company ever again.
Raising prices risks churn, reducing free tier risks lowering customer acquisition rate.
You can free tier but still sensible rate limits and QoS policy. You can have separate region or cluster for your paying customers. Finally you can spend time re-architecting GitHub Actions or just aquehire one of many competitors that do better co/cd.
Not remotely compared to github, and you can run GitLab OSS internally anyway.
If they are genuinely compute constrained on CI they should obviously reduce the free offerings rather than take down all their paying customers constantly.
Have you considered that you don't have the full context of the problem and it's not "just" a few small tweaks?
For example, Microsoft is known to pay significantly less than their competitors. I am guessing they just don't have the systems internally to say "This problem is worth literally billions to us in revenue per year, so we will pay millions to IC's to get it fixed and clear internal roadblocks".
It's not "just" small tweaks it is likely that people who are entrenched need to be moved and they need people with the right incentives in place to get it done.
Their c-levels wrote blog posts about the current state and why it happens, pretty sure they know what's going on.
I think the reason they can't make that change is that Microsoft is a slow moving rigid culture and doesn't have the kind of org that can adapt to a 10x or 100x increase in usage compared to companies like Google or the AI Labs that handled much more difficult scaling problems.
Have you seen this legendary email thread from 2003 https://www.osnews.com/story/19921/full-text-an-epic-bill-ga...? This isn't a new problem at Microsoft.
Having read many of the formal statements from the C*Os in the BigCos I've worked for, I can say with confidence that that is not at all a safe bet to be making.
It is also constant outages, and then it's your job to diagnose and fix it- and the technical work of that is the easiest part.
It's broken because a team figured out a new way to abuse the company's git infra to get around some process or limitation, and now you have to argue with their manager and then their director about why their team needs to stop using your VCS as a docker registry or whatever.
I'm assuming it's something to do with the USA users getting their first work in?
Or do you mean you want to see: "Something is wrong, we're looking into it."
I did work for a group using Atlassian bitbucket once, and that had its own problems at the time. I’m still pretty sure even a degraded GitHub beats it. That is to say I don’t know what alternatives for GitHub even are
Outages only come from our infra.
Migrating means migrating your ci workflows, but a decent llm can do that easily.
It is impressive they've been able to keep going, but this is a regular denial-of-service attack at this point, nonstop and relentless.
For example, GitHub can reduce/eliminate free GitHub Action minutes.
Or just cap concurrent jobs for free users from 20 to 2 or 1… lots of levers, their inaction is sad to see.
If you are doing CI/CD and want to be able to build and deploy on demand, you pretty much have to treat Github as a mirror and not the source of truth for your applications. If you have an outage, need to deploy a fix, and Github is down, you're on their recovery schedule, not your own.
GitHub also has more traffic, compare to GitLab
We've had very few noteworthy incidents self-hosting GitLab. We can size our deployments as we need, and perform upgrades during maintenance windows we control.
Stop.
> Gitlab Cloud
No.
Still grateful for a lot of the free services and the semblance of community that GitHub still offers. It does seem like it's still the yellowpages of tech.
There’s no CEO of GitHub to contact and Microsoft does not care as Copilot, and Tay continue to wreck the platform as the incidents continue.
No better time to self-host, since as this point your self hosted setup would be far most stable than GitHub as predicted 6 years ago. [1]