GitHub: Yesterday's Outage

github.com

14 points·timmorgan··6 comments

With all GitHub's server tech and architecture (described in fair detail in past blog posts), I'm usually surprised (and relieved a bit, I must admit) to see them fail at something.

Does anybody else announce outages with twitter and other public communications servers offsite? At darkmists.org, we learned a while back that we could not trust even our ISP to keep us accessible to the world. The iPhone allowed me to at least use twitter to update our players and attempt to set expectations. (Some of our administrators also try to keep tabs with players via IM programs, but that is not scalable.)

I think it's pretty common practice to host your status site separately from your site so that connectivity issues at your datacenter don't bring your status site down as well.

Engine Yard has a Wordpress.com blog (and they don't even have it as a subdomain of engineyard.com. I guess in case their DNS goes down). Not only does GitHub use twitter, but they also have status.github.com hosted on EC2. Heroku has a status site on Slicehost.

It just makes sense. Connectivity is a pretty common reason for downtime. Why would you want a point of failure that can bring down your site and your status site?

> Not only does GitHub use twitter, but they also have status.github.com hosted on EC2. Heroku has a status site on Slicehost.

Heh, that's a bit funny:

Github's hosted on Rackspace Cloud. They have a status page which is hosted on EC2 via Heroku. Heroku hosts a status page on Slicehost. Slicehost is... wait for it... run by Rackspace out of the same data center as Rackspace Cloud.

The main GitHub app is served from Rackspace's IAD data center on bare metal.

So, essentially, the cloud is now self-aware?

Reply on news.ycombinator.com