Evolution of GitHub's data centers | Hacker News Reader

Evolution of GitHub's data centers

githubengineering.com

115 pointsdoener22 comments

This is pretty generic -- isn't that how virtually any decent footprint site with physical infrastructure operates (at least from ~2000 onward)?

I'd be more interested in blog posts about either what they were doing before which was different/worse and got upgraded, or how they decided on specific things.

Their in-house tool gPanel is pretty cool, though (https://githubengineering.com/githubs-metal-cloud/) -- I've seen a lot of people build their own of that in ways which are far less sophisticated, or used one of the ~10 other tools for it. How much effort an org puts into making sure firmware is all the same is probably a good indicator of how competent they are, IMO.

>In memtesting we boot a custom MemTest86 image and monitor it while it completes a full pass. Our custom version of MemTest86 changes the color of the failure message to red which allows us to detect trouble. We’ve hacked together a Ruby script that retrieves a console screenshot via IPMI and checks the color in the image to determine if we’ve hit a failure or not.

Wow. Why is this done this way? Is there no other way to receive hardware memtest info? I would have thought that there was some way to revive machine serial output status in a datacenter.

Yeah, this seems pretty terrible compared to a modified memtrest that reports results.

Over network? Needs a networking stack in memtest.

Over serial? Physical serial is rare. SOL would work, but since it is connection-based you can't poll it.

This seems like a simple and reliable method to me.

Since they take a screenshot of the console they could output a qr code and read that.

That's sort of what they're doing. It's just a two-bit QR code: Red, and Not Red.

So, one bit ;)

Hah! Yes, totally. Haha I wonder why I said that?

There is also a "not decided yet" state. Hence, you were right, it needs two bits. :)

Patching the font colours are probably a few lines of diff, adding a QR code to a text-mode display... way more complicated.

Because its a couple line patch to memtest and a short ruby script. It obviously works well, why bother over engineering it?

Two things that stuck out at me:

- No mention of an European data center (data sovereingty and regulation might be something they don't care about, but I deal with it every day).

- No mention whatsoever of virtualization (although there was an older post mentioning a move to Kubernetes for a part of their stack).

I wonder what the economics of taking this to a public cloud provider would look like (full disclosure - I work on Azure), and how much they could do architecturally to benefit from that.

At a scale of Github’s it’s highly unlikely that this would be a cost effective maneuver - see Gitlab’s effort to move off of cloud providers as an example. There comes a point where you get so big that it no longer makes sense to use third party hosting. Dropbox is another example of this.

Gitlab isn’t moving: https://about.gitlab.com/2017/03/02/why-we-are-not-leaving-t...

Dropbox was primarily about moving storage off S3 which makes sense given their business.

Dropbox has run compute* in its own datacenters since 2008 or so.

*: That is, work that does not interact heavily with content

When you're using full racks of hardware for basically one product, what does a virtualization layer add?

Virtualization is useful for doing many small tasks on a single machine, especially if those tasks are owned by different companies (and you are ok with the tenant separation you get), but if you're using full machines it adds overhead.

1) They probably have a huge amount of data. Moving it without downtime might be an extremely costly and long process.

2) Cloud providers aren't generally that great for IO intensive application, I don't know if they use something like FusionIO cards, but if they do, it might be hard to get analog IO performances on cloud providers (i3 on AWS are quite good at that however).

3) If you can front the cash, manage the logistical aspects and have a relatively constant load and defined core functionality, bare metal is way cheaper.

Are they using Dell equipment?

Based on reference to iDRAC, certainly seems that way. Presumably they 1) don't have a huge amount of hardware (in the sense a commodity hosting provider or cloud provider would) and 2) get a decent deal from Dell.

I was let down that this wasn’t more of a visual experience.

Agreed. Their past post with huge lot of pretty pictures have been awesome.

Here are 256 photos from Stackoverflow hardware:

https://m.imgur.com/r/cableporn/X1HoY

Reply on news.ycombinator.com