Hacker News Reader

Comment by mattbeckman | Hacker News Reader

"With Amazon we experienced networking issues, hanging VM instances, unpredictable performance degradation (probably due to noisy neighbors sharing our hardware, but there was no way to know) and numerous other problems. "

Why do I get the feeling it was kind of a cop-out to just pack up and move without finding the root cause? I've seen it plenty of times: the "best solution" is to just find a different hosting provider.

In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code.

View on HN

> In my experience, I've never found an issue with an application on AWS that wasn't caused by either a misunderstanding of what was being offered (e.g. not provisioning enough PIOPS for database volumes), or simply issues with the application code.

You haven't been using Amazon long enough then.

Amazon is great for proof of concept. No upfront costs, extremely scalable, etc. Unfortunately, its expensive compared to physical hardware once you get to scale, and you may never solve underlying performance issues due to it being a shared tenant environment, even if you're a Netflix-sized customer.

You can use abstraction layers to isolate yourself from issues with the underlying metal. For example, I had a good thread the last time the maintenance reboots happened: https://news.ycombinator.com/item?id=9120289.

Solving multi-tenancy issues is hard, but not impossible. I think it's a lot easier with live migration. If a box is giving you problems, just move the load to a new box while maintaining the same IP addressing.

With respect to cost, yes, AWS gets expensive at scale, but if you're at scale your servers are generally not your major cost center (it's usually payroll and licensing).

That is what I like about the cloud. Running in your own hardware lets you be, well, "lazy" about application architecture. Running in a place where it is shared and instances can disappear forces you to design a lot more robustly and nimbly.

Of course, that design discipline is great wherever you are running....

Hardware is almost always cheaper than engineering time. Don't optimize prematurely.

Yes, but engineering is cheaper than running after real-time problems.

Now we get to the technical debt debate. Sometimes, you have to make good decisions now instead of perfect decisions later. The market doesn't care how elegant your code is.

The market cares about uptime, and cost relative to return.

Perfect should never be the enemy of the good.

> Perfect should never be the enemy of the good.

Only in a world where resources are infinite does this work.

No debate, I agree with everything you have said. But sometimes you have to clean that garbage up because it does matter to the market.

But doesn't Netflix successfully use AWS now?

Its possible they get special treatment if they are big enough (nobody else's jobs on their physical machines ... or something like that).

Netflix uses thousands of instances, they don't share servers.

Using thousands of servers doesn't mean that you don't share physical servers.

Even if you own all the VMs on a physical server it's hard to avoid the noisy neighbour problem. Netflix has its own instance monitoring tool - Vector - to handle these issues: http://techblog.netflix.com/2015/02/a-microscope-on-microser...

Yep, that is why I specifically noted in the article, that given enough resources, it is possible to survive and even thrive in the cloud (Netflix being one of the best examples of that), but in case of a startup it is not aways the best idea to keep burning money and engineering resources when your primary job is to keep building a business.

And no, I don't think they're getting any real special treatment from Amazon since every single talk from a Netflix engineer points out the the cloud-specific issues they're solving in their infrastructure software.

I've seen and fought issues with hanging and/or stalling EC2 instances, and the decision was made to move to physical hardware - it was one of those tradeoff choices you have to make about paying N dollars to throw money at hardware vs. M dollars at person-hours to investigate.

It's definitely kicking the can down the road (eventually you have to build such that failing infrastructure is transparent to your eng team), but I still think it was the right decision at the time. YMMV obviously. :)

I've had the odd problem with hanging/stalling instances before I learned to design for the cloud. Now those instances automatically get replaced with new ones and nobody would even notice if I didn't pipe the notice through to Slack.

Of course, this only works if whatever you use it for allows for this.

However, in most cases not using EC2 as a stateless throwaway computing resources is simply a matter of bad infrastructure design.

Reply on news.ycombinator.com