adpowers

Uhh, did you even read it? Under "Impact to Multiple Availability Zones", last paragraph:

"There are three things we will do to prevent a single Availability Zone from impacting the EBS control plane across multiple Availability Zones. The first is that we will immediately improve our timeout logic to prevent thread exhaustion when a single Availability Zone cluster is taking too long to process requests. … To address the cause of the second API impact, we will also add the ability for our EBS control plane to be more Availability Zone aware and shed load intelligently when it is over capacity. … Additionally, we also see an opportunity to push more of our EBS control plane into per-EBS cluster services. By moving more functionality out of the EBS control plane and creating per-EBS cluster deployments of these services (which run in the same Availability Zone as the EBS cluster they are supporting), we can provide even better Availability Zone isolation for the EBS control plane"

It is hard to test emergent behavior in large distributed systems, you pretty much have to actually run the tests live to see what is going to happen and see if it aligns with your predictions.

I don't think it covers authentication, but take a look at Thrift:

http://incubator.apache.org/thrift/

It provides an RPC framework and IDL.

Yeah, that looks more like a datacenter: http://goo.gl/maps/bYcG

Dead giveaway: the huge row of generators on the south side of the building. I'm having trouble identifying the other stuff because the photo is so overexposed.

Compare this to Microsoft's huge Chicago datacenter: http://goo.gl/maps/jVHw

My website and e-mail have been down for 12 hours and counting. Alas, my only instance that has survived this is running a website that is still under development.

I'm not sure it is a big deal if you expose both to Google. There is a setting in webmaster tools that lets you explicitly set which you prefer: example.com or www.example.com

http://www.google.com/webmasters/

Although, the preferred way is probably to use a redirect.

Heh, I know where you are coming from. In high school (before Google was really well known and popular) I participated in a "science olympiad". One of the competitions was focused on computers. You had to create a spreadsheet to calculate certain values, etc. One of the tasks was to answer a question using the internet. It was something along the lines of "Who was the man that refused to leave his house before threats of Mt. St. Helens erupting and was eventually buried in the ash." I typed something like [grumpy old man buried mt st helens] and the first result contained his name in the snippet. I finished way earlier than anyone else :)

Hmm. I had a very different experience. Most of the Google code I've seen them release is very, very high quality, like the Guava libraries: http://code.google.com/p/guava-libraries/

Well, if you are a large player like Dropbox you'll get really good S3 rates because you fall into the higher tiers. If you are in the 5+ PB tier you are only paying $13.2 for 20 GB/year, whereas Amazon is charging $20. If you have enough scale you can probably make the numbers pen out.

I'm uploading a ton of photos to test it out, then I'll delete them all. For free!

PS: Sorry for hogging your upstream Alan.

I'm curious: do you see any speedup from that Hadoop cluster? I always assume that on less than like 5 nodes it is faster to just run it all on a single machine unless you are highly disk bound.

Was it written that way in order to support future growth?

The master only job flow is designed to let users play around with the instance and discover things without having to pay for a full cluster. A single node versus multi-node cluster is configured way differently and that is why you can expand a single node cluster. If you had started with a two node cluster you would have been able to expand it.

Also, if you want Pig you should complain about it vocally on the EMR forum. That is the best way to get them to listen to you.

The hiring market is very competitive right now. It is hard for Amazon (or anyone) to hire good engineers. I think this is exacerbated by Amazon's lack of perks. My opinion is that if Amazon wants to hire the best of the best to work on AWS products they need to stop being so cheap ("frugal") and match the perks (and pay) other software companies offer.

This is very accurate of many teams within Amazon. Also, I don't know the exact story of what is happening with EBS now, but I have heard an increasing number of horror stories about EBS.

Hmm. I think you underestimate the effort that is spent on those two. RDS has really good replication which is really hard to configure and set up yourself. And having configured Hadoop I know it takes more than 5 minutes :) Perhaps Whirr makes that easier. Also, EMR's Hadoop is tuned to work really well with S3, which you don't get with stock Hadoop (or even with Cloudera's).