Hacker News Reader

Comment by davidjgraph | Hacker News Reader

Before the doom and gloomers come out, this is the first time since leaving beta I can remember it happening.

We left AWS about 18 months ago after one of the outages and switched to GAE. I've counted 3-4 big downtimes for AWS compared to this one on GAE. That's still a good decision (for now)....

View on HN

One thing to remember: this took down all of app engine for at least an hour. AWS has had only 17 minutes of downtime affecting all of us-east this year (that network glitch a couple days after PyCon) - the rest of it has been a subset of the service amplified by people rediscovering that they weren't as redundant as they thought.

The correct less to draw is that any one point of infrastructure is a risk, so you need to scale wide. This is possible to do with AWS regions, or other providers - even internal bare iron if you're so inclined, but impossible to do with GAE because you're committed to a single-vendor API as well as their infrastructure.

GAE applications are distributed across multiple data centers[1], so in theory you get "scale wide" automatically. Unfortunately it looks like there was some sort of flaw in the architecture. I believe this is the first systemwide failure of the HRD.

The real question is: Can you and your ops team build a "scale wide" system better than Google?[2] How much effort are you willing to put into it, when those development resources could be put into making features instead?

[1] For apps which use the high-replication datastore. Old (deprecated) master/slave apps are served out of a single datacenter.

[2] This is a reasonably serious question to ask. In GAE, you're one app among many, so a you-specific scaling solution might be easier and more robust than the generalized one Google builds. But not necessarily.

I completely agree that Google is likely to do better than many teams, modulo your second point (which I completely agree with - generic is much harder than specific). For me it really just comes down to the lock-in aspect: with GAE if you decide that Google isn't taking the platform in the right direction for your business you're looking at something close to rewriting your application. This is far from the most likely outcome - although pricing can be interesting - but it's the kind of thing you really want to consciously acknowledge, similar to the way Netflix has accepted the risks of AWS by investing heavily in failure management tools.

I keep hearing the lock-in argument over and over and I'm not quite sure that there is a solid basis for it beyond a level of paranoia. It's fine to be paranoid, I just don't want it to hold me back unnecessarily.

Looking at things more closely, the only thing that you're truly locked into with GAE is the esoteric nature of the datastore. This isn't any worse than picking say MySQL vs. Oracle or Riak vs. Mongo. Most applications end up depending on some sort of specific functionality of each database that they are written for. While it would be difficult to migrate to another solution for storing your data, it wouldn't be impossible.

There is no way to predict what direction any product might take in the future. Look at the way that Oracle is treating MySQL now. Tons of vendor lockdown there.

The only reason to migrate away from GAE would be if you find out that your application doesn't work well on it (pricing, scalability, etc) or if Google decides to kill GAE entirely. Hopefully you do the analysis of your application before you decide to use GAE (ie: you can't blame GAE for you deciding to use it) and with a 3 year deprecation promise, I'm pretty confident that it will be around for a while longer.

> Looking at things more closely, the only thing that you're truly locked into with GAE is the esoteric nature of the datastore. This isn't any worse than picking say MySQL vs. Oracle or Riak vs. Mongo.

Sure it is. If you pick the wrong DB your only locked into that DB. You pick GAE and you're locked into GAE's DB.... and GAE. I can move my MySql db to another cloud provider.

Your analogy of moving to another cloud provider is incorrect. Unless you've been 100% database agnostic, you can't just migrate your schema and data from say MySQL to Postgres.

With the datastore on GAE, you can get your data out of it and move it into something like MongoDB. I'd argue that it would probably be less code to move to MongoDB because the datastore has all sorts of esoteric issues that you have to code around (like the way that entity groups and transactions are handled).

In terms of the rest of your application, it is just a standard webapp in whatever platform you choose. The .war file I have for GAE will run just fine in Tomcat. The only real lockin is the way you store your data.

@latchkey I think you are missing the point the MatthewPhillips is making. With GAE's DB there is nowhere to move, so if you want to move you would have to migrate your data to a different DB engine. If your hosting/cloud provider uses something standard like MySQL you can find another provider or roll your own if you decide to migrate out.

> The only reason to migrate away from GAE would be if you find out that your application doesn't work well on it (pricing, scalability, etc) or if Google decides to kill GAE entirely. Hopefully you do the analysis of your application before you decide to use GAE (ie: you can't blame GAE for you deciding to use it)

This seem too simplistic. Off the top of my head, you may leave because:

* you want to do something new in your app that is not possible/effective to do within GAE * you find another option that is cheaper * you find that some of the assumptions and architectural decisions you took on day 0 no longer hold * you did the initial analysis, and it was wrong

I do agree with GAE being awesome and lock-in not being so bad, but I doubt the "you have it all figured out before building it so it's going to work forever" idea.

FWIW, I frequently want to do things that are not possible/effective on GAE. And so I do it - several of my GAE apps communicate with services I set up in rackspace cloud (30-40 ms of latency away). It would probably be even lower running in Google's cloud service, but I haven't gotten an invite yet.

GAE is not "all or nothing". You can still run exotic services in other hosts. Or, for that matter, use GAE for specific services in your "other cloud" app. You get two bills. Not much of a downside.

With the Java API you can write code that is standard (JDO/JPA for database stuff), that works across any Java environment. So it doesn't lock you in if you write your code in a smart way.

Except that writing code using JDO/JPA is a nightmare, at least on Datastore. So that's almost certainly not the smart way. I'd rather rewrite my app three times - go with Objectify (simple, easy, maps well to Datastore concepts) and just deal with the rewrite if it happens.

> you're committed to a single-vendor API as well as their infrastructure.

Not quite. You can roll out your own App Engine platform with AppScale:

http://appscale.cs.ucsb.edu/

Sadly, TyphoonAE (http://code.google.com/p/typhoonae/) seems abandoned.

It's one of the first times the whole service has been down, but parts of the service go down at least once a week. memcache, task queues are "elevated" with regularity and urlfetch is frequently down totally. ("elevated" generally means unusable).

Of course master/slave even has scheduled downtime.

Yep. We've never seen anything like this before.

Same. Minor updates and issues pop up with GAE but I've never experienced an outage such as this in my two years of use.

Wonder what's happening.

I have yet to experience downtime with RackspaceCloud and I've been using them for like 3 years.

Me too.

I had network issues with my two virtual machines on RackspaceCloud about a year or so ago (something was wrong with their routing). They were resolved quickly, though.

Reply on news.ycombinator.com