Hacker News Reader

Comment by chesser | Hacker News Reader

To me, that sounds like encouraging a race to the bottom.

When you're in the business of storing other people's data, things like transactions, binary backups, QUICK RESTORES, and separating dev from production shouldn't be afterthoughts. They are core attributes.

I would understand this for a demo, but not after a couple of years and half a million users. They have paying customers.

"Reliable code hosting

We spend all day and night making sure your repositories are secure, backed up and always available."

View on HN

The half a million users validates their techniques, no matter how much armchair quarterbacking.

The code repos were never in danger, and they've been killing it in the market because they are racing to add awesome extra features, not racin to the bottom.

I don't disagree that they can do more in their db operations, and that it's fine for us paying customers to demand more, but the reality of startups is that it very much is a race and features, for availability or recovery or other, are viciously prioritized and many things don't happen until something breaks.

If you don't thing Github cuts the mustard, svn on Google Code probably won't have problems like this...

Disclaimer: I'm personal friends with much of the Github crew.

> The half a million users validates their techniques,

Let me introduce you to my friend, GeoCities.

Funny thing about free hosting.

> no matter how much armchair quarterbacking.

Transactions aren't armchair quarterbacking. Binary backups aren't armchair quarterbacking. Separating development from production isn't armchair quarterbacking. Kindly, you have no idea what you're talking about.

You can't give them credit for the repositories being mostly intact, when the ONLY parts that broke were the parts they mucked with to tie them into their database.

> but the reality of startups is that it very much is a race and features, for availability or recovery

And those are exactly the places where they screwed up.

Are you sitting in a chair? Do you not work at Github? Then you're armchair quarterbacking. We all are. Even if you are a quarterback for another team (I'm a DBA myself)

Anyway, it's a great discussion so we can learn from other peoples mistakes. I will be triple checking my restores later today, and likely halt the project I'm working on to get cold standbys shored up asap. (but it's hard to prioritize house keeping over customer centric features in the race I'm running along side the Github crew)

"a pejorative modifier to refer to a person who experiences something vicariously rather than first-hand, or to a casual critic who lacks practical experience"

Ironically the most fitting application would be to say that they're armchair quarterbacking their own database administering.

Unless you work for github, you lack practical experience in github's internal operation.

Fortunately, there is this thing called "science" which means we can understand things about the world regardless of where we live. As Dawkins would say, there is no such thing as "Chinese Science" or "French Science", just science. Similarly, there is no such thing as "Github MySQL" or "Github separation of production and development systems" in that same sense.

These are categorical mistakes.

    "If you don't thing Github cuts the mustard, svn on
     Google Code probably won't have problems like this..."

Neither will Mercurial on Google Code ;-)

Disclaimer: I work for google on the Project Hosting product.

Quite fair! Google code is sweet

If we're going to use hindsight, we might as well look at the cause and the effect.

Yes, they missed out some pretty obvious things. And what happened? A few hours of downtime because their restore was slow, plus a tiny bit of inconsequential data loss. Hardly a catastrophe.

The fact is that every site has some sort of problems. Many of them will be completely obvious like this one. And while github could have gone through and attempted to fix them all, I much prefer they spend their time doing things that have more than a few hours of impact on my life.

No doubt they will fix the issues involved today.

Your post is a good illustration of the type of attitude we want to scrupulously avoid when appointing someone to be in charge of a database.

Reply on news.ycombinator.com