I see no evidence here that EC2 hardware is "changing underneath" anyone; rather, some EC2 instances are running on different hardware from other EC2 instances, and PiCloud is moving the customer between different types of hardware.
We did not intend for readers to think that hardware configurations were changing while the instance (virtual machine) was running. As EC2 does not support live migration, this is not possible.
When you use EC2 (or any cloud infrastructure) a lot, you stop thinking within the constraints of physical servers. The very nature of EC2 is the first letter of its acronym: Elasticity. You are able to bring up functionally equivalent duplicate servers when the demand arises and terminate them when the demand no longer exists. This exists not only for HPC applications (our domain), but even horizontally scaling web servers based on demand.
EC2 allows you to request different "instance types". A given class offers the same cpu speed[1], memory, and disk usage. To make scaling tractable, you need to make the assumption that if your program operates correctly on a single (or even a few), say, m2.4xlarge instance types, it will do so on all of them.
Customers do not use us solely because we provide an excellent distributed computing interface. They also use PiCloud, because we (thanks to the capabilities of Amazon EC2) are highly elastic. This customer doesn't want to continually run several dozen m2.4xlarge equivalent instances, because their utilization doesn't justify doing so. Instead, PiCloud allows them to run massive computation for a short time and not incur charges when the servers are no longer needed. (Resulting in far lower average costs than having their own on-site cluster.)
Think about how this customer would have used EC2 directly within these constraints. Their workload would be something like deploy a few dozen servers, run computation on said servers, and terminate servers when complete. After getting this application to work correctly, they may conclude they are "done". It might work for months, and then one day, he would find that some "jobs" raised Illegal Instructions. And there would have been no way to anticipate this happening. And from their perspective, where the "requesting dozens of servers" is part of the application, the hardware had changed underneath them.
[1] Yes, performance will differ a bit across different instances. However, you expect that functionality is identical across all instances of the same type (e.g. m2.4xlarge).
Basically the picloud model is you do a "cloud" call, and that runs on some instance (usually shared, picture a single instance doing some number of picloud calls in parallel) at some point in the future.
The sticking point for them was that the same instance type was backed not only by different CPUs, but by some subset of instances whose CPU capability advertising was broken.
It does actually break "instance type equivalence" for people running real code. I think Amazon will fix it though.
So picloud stripes (simplifying; I imagine it's a pull model with some load balancing smarts) python "cloud" calls across a variety of instances they have warmed up, but there is an assumption that, say, arbitrary Cython code will work the same way between one m1 instance and another.
That turns out not to be the case. Some (say m1 for argument's sake) instances on Amazon are broken because they advertise the wrong CPU capabilities. So not all e.g. m1s are alike...
We don't even know that the case you are talking about, of stopping, then starting an instance leads to it living on different hardware types, though I would suspect that is true.
TFA is saying that when you boot NEW instances you sometimes get subtly different hardware, which when deploying binary code can lead to issues.
That does seem like a valid EC2 bug, but also not one that most of us will ever run into unless you are doing cool massive dynamic scaling like they are.
The best part of that day was a guy from the government who gave a great talk on his experiences virtualising from running separate physical servers. While he was very much in favour of it, he mentioned a few drawbacks that the vendor talks obviously played down, but probably the best thing he said was "when doing something major with infrastructure like this, get your boss at the top on board early, because everyone above you will try to scuttle it to CYA if it goes wrong."
[1] http://www.vmware.com/products/datacenter-virtualization/vsp...
http://kb.vmware.com/selfservice/microsites/search.do?langua...
The original article just struck me as yet another company that doesn't know how to properly run applications in virtualized environments. Yes, if your hosting provider supports hot or cold migrations, you should be aware and develop accordingly. I do see this quite often, that still today, people are surprised that their VM is not always running on the same physical box at all times.
The problem they have is that they are trying to support arbitrary Python (which includes underlying libraries like LAPACK which depend on runtime CPU detection working correctly) and expecting code will work the same across identical instance types.
I think normal people would consider this a fair assumption, and in the event that AWS's advertised CPU capabilities were not broken for some instances, it would hold.
I don't think it's fair to say that they're ignorant of how virtualisation works. Disclaimer: I am a huge picloud fan and they have saved me a lot of time.
This has nothing to do with not running on the same physical box and more to do with some physical boxes are advertising incorrect instruction sets to the OS.
With that said, you can "migrate" EBS-backed instances by start/stopping them which (may) place them on different hardware Indeed, if you are large enough, you'll occasionally get emails from Amazon directing you to do just that due to scheduled maintenance on the physical machines.
If you stop your ec2 instance then you are effectively creating a whole new vm instance when you start it. That's not exactly a secret or something unexpected. It's why ephemeral storage is wiped clean and why you can only retain data via ebs (which is not local storage).
If you restart an instance (rather than start/stop) then you stay on the same hardware.