GitHub Incident with Webhooks, Actions, etc. | Hacker News Reader

GitHub Incident with Webhooks, Actions, etc.

githubstatus.com

81 pointsjbredeche61 comments

So frustrating. You can only experience so many of these outages before you seriously have to start questioning whether it makes sense to continue to use GitHub (and many have already long since reached this point)

We switched to self-hosted GitLab several outages ago

Yeah, switched over to self-hosted Gitlab back in mid-June and feels like the right choice

Same here. Gitea has been pleasant to use for my business, and I've had 0 downtime issues compared to near-daily problems with GitHub during the first half of the year.

Edit: Sorry, thought you wrote Gitea rather than GitLab. I've also had a nice experience with self-hosted GitLab years ago, and I'd wager that pretty much any self-hosted option would be better than GitHub at this point.

Just don't expose them to the internet. They have a lot of CVEs latey, some even allowing full control

How would that work out for remote workers and agents?

VPN or tailscale or client certificates on https endpoint.

> several outages ago

So you switched last week?

self hosted forgejo and forgejo action runners for my homelab

Yup am running too for 2 years now and love it. It’s my favorite service. Uptime is basically 100% since it’s only down when I’m doing server maintenance (apt get update and reboot) and then I’m not pushing anyway.

I implemented my own, it was time well spent.

The features I care about work as I want them to, the features I don't care about are kinda janky.

I'm now running the actions elsewhere, but yeah, I'm almost ready to self-host a full alternative.

Forgejo with a self-hosted runner is amazing.

Isn't this an example of how everyone wants to believe that vibe-coding software is enough, but no one actually wants to depend on vibe-coded software (like GitHub)?

I switched to Forgejo and it's such a breath of fresh air. You can feel the solidity, snappiness and dependability of a good old carefully coded app.

Except Github outages are not related to vibe coding Github itself, and the current situation is arguably only tenable at all because they're able to rapidly add optimizations and adjust their architecture.

If you've been paying attention to any news related to this [0], they've been drowning with the rise of agentic development. By September this year, they were up to six times the monthly activity YoY. Even after moving to Azure, they're struggling to keep up with the sheer amount of traffic being thrown their way. But they have so far been adamant on not gutting features like free actions pipelines for public projects.

[0] https://github.blog/news-insights/octoverse/octoverse-commit...

> Even after moving to Azure, they're struggling …

More like _because_ they moved to Azure. Azure has been nothing but crapulent. GitHub needed more capacity, but they chose the wrong place to add that capacity.

They weren't given a choice, as I understand it.

Maybe nobody under whatever MS VP is in charge of GitHub, but somebody at some level had a choice, and they chose poorly.

At this scale, not using the likely at-cost infrastructure of their parent company would surely have caused revenue problems, no?

How do we know that it's not related to vibe coding GitHub?

There can be multiple reasons. The flood of agentic development repos is real. But showing another reason doesn't show that vibe coding GitHub isn't a reason itself.

Yes, but it could also be because they're busy eating babies.

Let's just assume Occam's razor and give them benefit of a doubt over the six-fold usage growth figures.

Six times! Oh the tragedy! The only thing I remember being down more was Twitter circa 2009-2010, but they grew 15x in tweet volume on a hilariously inadequate RoR platform.

This was back when the 10k challenge was still a real thing for web servers.

GitHub’s UI is slow and clunky. And that has nothing to do with how many vibecoded lines of code are being pushed.

> And that has nothing to do with how many vibecoded lines of code are being pushed.

I think it has to do with the grandfathered code they had to replace to make things more efficient. And it's way more faster than 5 years ago, come on!

I feel like this is a very unserious response. Github is enormous and were at capacity in April at 2x YoY, then they tripled their traffic again within a few months. They have added an additional 4 million CPUs and are still struggling to keep up with an insatiable, exponentially rising demand.

As of September, they handle 2.8k commits and 1.4k actions a second - each which involve spinning up docker, installing hundreds of MBs of dependencies, and then running things like CPU-heavy linters.

If a blog with 100 people goes up 6x, that's nothing; a service with 150,000,000 active accounts going up 6x is a big deal. And I know you know that.

> GitHub’s UI is slow and clunky.

Can you read TFA or just not participate in the discussion, please? This has literally nothing to do with the UI.

Isn’t part of CS fundamentals learning to scale beyond 6x ?

Any dev worth their code thinks vibe code is enough? i think we are all just doing what management demands

Still mid switch to self hosted Gitea here!

gotta get on tangled.org!

That time has long passed IMO. This time last year GitHub started having these outages (before the explosion in agentic use, and shortly after the announcement they were moving everything to azure), and by February/March it was clearly untenable IMO.

I visit https://status.github.com it is the second suggestion, after https://github.com/ itself.

When I type git in chrome, it autocompletes as githubstatus.com.

Someone needs to create isgithubdown.com

You mean: https://mrshu.github.io/github-statuses/

And for the e-ink display hanging on the wall: https://www.dayswithoutgithubincident.com/

We self-host actions and this still impacted us. Incident resolved 30 minutes ago, but all our builds are still queued/pending. Maybe GitHub needs to replay some webhooks?

Polling wins yet again.

For those using GitHub Actions who want to be out of the blast radius of these outages, we at Depot built our own CI engine that speaks a GitHub Actions syntax but is our own control plane and runner [0]. Connected to all of the other things we do to accelerate CI, builds, tests, etc.

Happy to answer any questions or hook anyone up with an extended trial that wants to move things over to it.

[0]: https://depot.dev/docs/ci/quickstart

There has been degradation in their copilot cloud agent in the last few weeks (sessions erroring/disappearing, connections not working, etc) that is not on a status dashboard.

Dead in the water for actions that are merge requirements here.

I've believed, since jenkins was the the new hotness, that you should ALWAYS have a path to prod outside your CI. You can put whatever restrictions you want on it, but it must exist and be tested from time to time.

Even after they “resolved” it. I had two stuck actions in the queued state. Canceling failed. They are still stuck as “Queued” 2 hours later.

We have tons of similarly queued runs and missing checks that we get to go sort out, but the incident for them is resolved lol

Codeberg has been pretty stable for me

Getting used to this!

The more these incidents happen, the more I think about I talking to investors about investing in my GitHub alternative; an old computer in the corner running Gitea.

I was setting up a project for CI/CD for the first time in a much more complex way than before and this issue had me scratch my head a lot as I was questioning my sanity. Then I remembered it's GitHub and that it's just another outage, just not reported on their dashboard yet.

After the GitHub post about their new git storage architecture, I wonder how other parts of the system, like issues, actions and webhooks, are being improved to handle current scaling problems, if at all.

Apparently the git layer is not the only thing that needs a little attention.

I don't know them, and am not an investor, but I'm trying Blacksmith as a GHA alternative because I'm losing my mind over the regular downtime. Apparently all the cool kids already use it, so... Obviously this doesn't help at all with Webhooks.

update: first jobs are green already. migration took 5 minutes. so far so good.

Yes Blacksmith is great. We use them for all our CI jobs. Make sure to enable the Docker build cache if you use it.

In our benchmark, Blacksmith was the fastest runner among the providers we tested (Warp was another one) and then the build cache adds additional time savings.

Wonder how soon GitHub will tighten rate limits on OSS GitHub further and start charging per API call like x.com did to weed out bots...

Not being able to isolate OSS and Enterprise workloads properly is also quite the downer.

If you believe Twitter "weed[ed] out the bots" I have a bridge to sell you.

I'm not saying bots/LLMs aren't making GitHub's job harder but I saw an uptime graphic with a pretty stark and obvious change pre/post acquisition by Microsoft.

You can buy Twitter accounts in bulk

https://hstockplus.com/accounts/twitter-x-accounts?sortBy=pr...

i have githubstatus.com on my bookmarks. Maybe i should make it my homepage?

I have been moving everything away to a self hosted Forgejo, and I am so happy.

But now I wonder about Netlify. Their dashboard is jammed full of prompt boxes for agents. Could more AI psychosis there turn that into a house of cards as well?

I'm curious where you're seeing these in Netlify? Maybe I'm desensitized and ignore them now.

Again? Last time it went down was less than a week ago. GitHub continues to remind you to self host. [0]

6 year prediction and counting. [1]

[0] https://news.ycombinator.com/item?id=50307843

[1] https://news.ycombinator.com/item?id=22868406

Great day to be on Codeberg

Does Codeberg make sense for enterprise customers? I thought they were for OSS only?

No. And I think there was even an outage sometime recently as the single(!!) server they run it on was down for some reason.

Another, another incident

Yet everyone still uses GitHub lolz

Reply on news.ycombinator.com