I'm not very familiar with Moz, but it looks like they have an analytics product, which is probably particularly sensitive to slowdowns.
The cloud market is maturing now and a hybrid private/public model I think is going to become more of the norm, for companies that are large enough that this makes sense.
It is cheaper to run your own hardware when you are big enough, and then use AWS for failover / load spikes / cool tech, etc.
About two thousand dollars a month gets half a rack with a high-speed, highly-reliable connection in co-location. The lease payments on several extremely high performance, flash storage rack servers would be a couple thousand dollars a month. Throw on some shared nothing redundant virtualization, and you have the equivalent of an army of 4xmegaxlargemongos.
AWS has a place (such as being the redundant offsite backup for that scenario, or for extremely variable service requirements), but endlessly on here we see people disbelieving alternatives make sense.
But you don't come to AWS just for EC2 instances. You're there to combine their portfolio of services in various ways. Because of this, it'd probably be wrong to just compare $1000 on AWS vs what you are paying now.
Technical details:
I don't know exactly how they do it but my guess is that they are I/O bound on the crawling side, then CPU bound on the parsing and processing side. I'm assuming they use different machines for those tasks.
On the crawling side, the index with 60-70 billion URLS was using 80 cc2.8xlarge machines with a backup on 200 c1.xlarge machines. (http://moz.com/blog/one-step-back-two-steps-forward)
(DISCLAIMER: I'm in the same industry. I am a developer with working for an indirect competitor. I know a few engineers and non-engineers who work at Moz.)
I would imagine the main bottleneck, by far is the I/O with reading/writing to the database clusters and search index. Crawling is relatively cheap, it's storing that data that's hard.