I don't have the experience with large AWS setups to comment about the cost implications, but improved reliability and efficiency strikes me as questionable - AWS's reliability has improved markedly over the last 18 months to the point I cannot remember when their last major outage was.
I'm keen to hear other peoples' thoughts on this..
I'm not very familiar with Moz, but it looks like they have an analytics product, which is probably particularly sensitive to slowdowns.
The cloud market is maturing now and a hybrid private/public model I think is going to become more of the norm, for companies that are large enough that this makes sense.
It is cheaper to run your own hardware when you are big enough, and then use AWS for failover / load spikes / cool tech, etc.
About two thousand dollars a month gets half a rack with a high-speed, highly-reliable connection in co-location. The lease payments on several extremely high performance, flash storage rack servers would be a couple thousand dollars a month. Throw on some shared nothing redundant virtualization, and you have the equivalent of an army of 4xmegaxlargemongos.
AWS has a place (such as being the redundant offsite backup for that scenario, or for extremely variable service requirements), but endlessly on here we see people disbelieving alternatives make sense.
But you don't come to AWS just for EC2 instances. You're there to combine their portfolio of services in various ways. Because of this, it'd probably be wrong to just compare $1000 on AWS vs what you are paying now.
Technical details:
I don't know exactly how they do it but my guess is that they are I/O bound on the crawling side, then CPU bound on the parsing and processing side. I'm assuming they use different machines for those tasks.
On the crawling side, the index with 60-70 billion URLS was using 80 cc2.8xlarge machines with a backup on 200 c1.xlarge machines. (http://moz.com/blog/one-step-back-two-steps-forward)
(DISCLAIMER: I'm in the same industry. I am a developer with working for an indirect competitor. I know a few engineers and non-engineers who work at Moz.)
I would imagine the main bottleneck, by far is the I/O with reading/writing to the database clusters and search index. Crawling is relatively cheap, it's storing that data that's hard.
Moving to your own hardware lets you actually reason about what's going on in a controlled environment, and at the scale that Moz is working at allows us to get hardware that's robust, well-supported, and reliable enough to have a pretty serious net positive overall.