jread

It was a multi-region, multi-hour outage on Nov 18. To my knowledge AWS has not experienced a multi-region outage. That said, they are learning from these events and improving stability.

It's much better now. Check out the new Ivy Bridge/Haswell, SSD backed D and G series VMs. They are a vast improvement over the old AMD/magnetic backed A series.

I wrote that post - new GCE capabilities like pd-ssd and local SSD are very competitive for price and performance. You can review current benchmark metrics here: https://cloudharmony.com/benchmarks

We aren't able to differentiate - Rackspace stats include both cloud and dedicated.

Some quantitative data - we monitor Alexa and Fortune 500 marketshare for cloud services. This article is mostly accurate - Google has not taken much marketshare according to our stats. Currently, EC2 is 13% for Alexa 1k/10k and 11% Fortune 500. Rackspace is 5.5% Alexa 1k/10k and 19.4% Fortune 500. Google is under 1% for both, while Azure is between 1-2%. Our stats are available here:

http://blog.cloudharmony.com/2014/12/compute-marketshare-ale...

https://cloudharmony.com/cloudsquare#compare-aws:ec2-and-goo...

Edit: added blog link

But you're sharing the 1Gb/s link with who knows how many $5 droplets - resulting in unpredictable IO. Linode network IO is dedicated and scalable to 10 Gb/s. DO is shared and capped at 1 Gb/s - even on the largest droplet.

Linode is a more capable, performant and reliable platform. e.g. Look at a 90 day availability comparison: https://cloudharmony.com/status-3months-for-linode-and-digit...

My lease is up 18 month - next car is a Model X

Its good for around town driving, stressful for longer trips due to varying range (50-80 miles). Biggest complaint is because it is FWD, and the battery in back, traction is terrible going from a stop up hills, or on wet surfaces.

Thank God, I own 2 Fords including a Focus Electric. Sync has been by far the most painful part of ownership. Unintuitive, slow, buggy, flakey bluetooth - just a terrible user experience.

Great to see some fresh ideas in this space. NSONE filter chains are a nice take on what other services implement as disparate features (directional DNS, load balancing, health checks, ...)

I don't recall an AWS outages of this magnitude. Most cloud outages, including AWS tend to be data center/AZ specific.

I run a site that monitors cloud service availability. Based on VMs and Blob storage containers I maintain and monitor, the outage affected every US Azure region with 1-2 hours of downtime: https://cloudharmony.com/status-for-azure

We use AWS Elastic Beanstalk for Docker deployment. Can you comment on how this is different? I see it supports multiple containers per VM - are there other significant distinguishing factors?

AWS released PaaS docker support 7 months ago: http://aws.amazon.com/blogs/aws/aws-elastic-beanstalk-for-do...

Linode

NY2 appears to be down (again) https://cloudharmony.com/status-for-digitalocean - other regions are still up

Appreciate the constructive feedback. Just a few points of response:

1a) Core selections were not arbitrary - they covered most compute instance sizes offered by each service: 2, 4, 8 and 16 cores

1b) Testing was not skewed by faster cores. CPU performance analysis was based on SPEC CPU 2006 using base rate runtime. SPEC CPU is a multi-core benchmark with metrics that scale linearly on number of CPU cores.

1c) The value analysis (based on CPU performance and price) highlights differences in pricing between services. I believe matching CPU cores and deriving value is preferable to comparing compute instances based on price.

2) The intent of the post was to provide and compare multiple relevant performance characteristics for these types of workloads: CPU, memory, (non-cached) storage IO, network. Each of these characteristics is analyzed separately using relevant benchmarks and runtime settings. If your workload relies more heavily on cached IO, then more emphasis could be placed on the memory performance analysis.

3) There are hundreds of cloud services. High level analysis like that provided in this post can at least get users pointed in the right direction.

4) Web server network performance analysis is based on "eyeball" or last mile testing from our cloud speedtest http://cloudharmony.com/speedtest. As stated in the post, this is the most complex performance characteristic to measure. Internal network performance is also included in the DB server analysis.

Good point - user experience may include cached and non-cached I/O so it would be beneficial to include both in this type of analysis.

The benchmarks binaries, configurations and runtime settings were generally consistent for instance types of the same size across services, but we didn't verify efficacy of the benchmarks as they ran.

Not really for latency - throughput is more variable between instance type, particularly for services like Rackspace that have instance specific limits.

Point taken - graphs will be improved next time.

Linode will be in the next report. We chose the services and began working on this report before Linode announced their infrastructure upgrades.

Mean latency RSD was EC2: 8.9%; Azure: 10.3%; DigitalOcean: 23.7%; GCE: 8.9%; Rackspace: 6.1%; SoftLayer: 16.8%.

It is possible - but it would be difficult to optimize for every performance characteristic because they are derived from many different benchmarks. This was a problem in the tpc-c years with database vendors optimizing specifically for a single benchmark.

The intent of the fio testing was to measure block storage not cache I/O. direct I/O was set in fio configs, but is usually not honored by the hypervisor. During run-up a 100% fill test was performed using refill_buffers and scramble_buffers to break out of cache. Then optimal iodepth settings are determined for each workload and block size by running short tests with incrementing iodepth settings (targeted for maximum iops). Once iodepth is determined, 3 iterations of tests are performed, each with 36 workloads (18 block sizes, random + sequential). Each of these is 15 minutes (5 minute ramp_time, 10 minute runtime). Since asynchronous IO and variable iodepth settings were used, latency wasn't compared. Total test time per instance for run-up and 3 iterations was about 36 hours. fio configs are available here (iodepth and device designation are added at runtime):

https://github.com/cloudharmony/fio/tree/master/workloads

Good feedback, thanks - just added a link to the end.

GCE no longer offers a local storage option, this is one reason slower IO. They also cap IOPS for better IO consistency.

The SPEC CPU 2006 estimate comment is a legal requirement for any public results that cannot be guaranteed to be repeatable (which will never be the case in a virtualized, multi-tenant cloud environment). The results provided are based on the mean from multiple iterations on multiple instances from each service. If you were to run it on these same instances your results should be similar. The post also provides standard deviation analysis for all SPEC CPU runs on a given instance type to demonstrate the type of results variability you might expect.

A t2.medium starts with enough credits for about 15 minutes of 2 core CPU saturation and accumulates 12 minutes/hour thereafter. In non-burst mode it is about 5X slower. For this type of workload you'd likely be better off with a c3 or m3. t2 is a better fit for long term usage with periodic spikes (20% or less of total operational time).

Amazon offers a full ecosystem of services - compute, object storage, CDN, DNS, DbaaS and a lot more - DO and Linode are VPS services with hourly pricing.