Hacker News Reader

Comment by DanBlake | Hacker News Reader

Using EC2 for anything but overflow is silly. The costs are extraordinarily high if you are doing decent volume ( read- if you have more than 3-5 servers )

Bandwidth is ridiculously expensive with them. You can get 3-4x cheaper per megabit going dedicated.

Servers are crazy expensive. Compare the most powerful machine they have vs something on 10tb / gigenet / theplanet for the price. You will definitely end up with a more powerful machine for half the price on either.

The only real advantage to using ec2 is the hourly billing. Make its perfect for overflow, but thats about it. I read about alot of startups that use ec2 for things like webservers or other servers that have 100% reliance.

Dont get me wrong. Some of Amazons offerings are great. The CDS and S3 can take the bullshit out of dealing with the complexitys of each and might save you a network engineer or 2. However, for EC2 there is no excuse. Its no more difficult to setup a normal dedicated server than it is a EC2 box.

/endrant

View on HN

I agree it's more expensive, but I think you are undervaluing the instant provisioning capability EC2 gives you and overestimating the impact of the increased cost.

The way I look at it is if all these services were free, EC2 would win based upon features. So, the question is, does cost matter?

For us, a single engineer's salary costs more than our entire yearly EC2 bill. And, we're not even using cost saving measures like elasticity and reserved instances.

And, EC2 actually drives costs down in ways hosting does not. First, as you mentioned, it dramatically reduces the need for IT staff. (Though it doesn't eliminate it, of course.)

Second, and IMHO, more importantly, the constraints EC2 imposes on you forces you to build fault tolerant, shared-nothing systems. I'm sure many folks here have worked at companies with hosted servers to find machines that are chock full of random cron jobs, services, and so on. The instant provisioning capability of EC2 combined with the constraints makes it very hard for you to have these one off "god boxes" your system is hinging on. Beyond that, as things get too messy, you have the freedom to automate and shut-down and bring up fresh nodes instantly with no increase in cost. It's hard to me to estimate the amount of man-hours we've saved and disasters we've avoided due to this capability. It's allowed us to cleanly "refactor" our entire cluster over time since we can reprovision nodes on demand as we improve the services or architecture on those nodes.

"The way I look at it is if all these services were free, EC2 would win based upon features. So, the question is, does cost matter?"

I think this is the important part of the debate. Many people don't use these features, but with EC2 you are paying for them regardless. For example, you pay for the ability to pay by the hour (in terms of increased costs), even if you never plan on doing so. Yes, reserved instances can help, but they don't fully cover the cost difference.

So it really does come down to what features you need. If you do use all the features, I'd imagine that EC2 is a great deal, because everyone else is subsidizing your use of them. If you don't want any of the features, well, then you're subsidizing everyone else.

I think the "cost shouldn't matter" argument isn't relevant. If cost doesn't matter, it is hard to make a case for EC2 vs spending the money to develop your own infrastructure + hiring lots of people + vastly overprovisioning your own datacenters.

In my experience, this is true and extremely valuable:

"the constraints EC2 imposes on you forces you to build fault tolerant, shared-nothing systems."

EC2 forces you to care immediately, not later when the shit hits the fan (which it always does, eventually).

I dont buy that ec2 relieves the employee need. Its just as difficult, if not MORE difficult to setup a ec2 instance than buying a dedicated server.

S3, SDB, CDS, etc.. Yes, those make things easier. EC2 Does not.

"Its just as difficult, if not MORE difficult to setup a ec2 instance than buying a dedicated server."

Is this from experience? If so, please share it.

Sure. I use EC2 to handle our overflow during peak hours. I had to literally look up guides to dealing with it. Its not exactly straight forward how the IP system works, and figuring out all the ec2 specific crap, images and storing the image in S3, etc. etc.

I'm not sure I buy this argument. You had to look up guides for how to set up something you are using because you find it valuable. How is this any different from any other tool you've found valuable and set up? I'm sure that too seemed arcane at first.

Oh well, the old cost argument again. As a satisfied EC2 customer [who also happens to be fully able to use a calculator] I'm happy to take these apart, again and again.

Bandwidth is ridiculously expensive with them. You can get 3-4x cheaper per megabit going dedicated.

Yes. But that does not normally matter. Bandwidth has become ridiculously cheap either way; a terabyte sets you back a measly hundred bucks on amazon. And if you're pushing more than a few dozen of them then most of it is very likely going over a CDN (or a set of separate dedicated servers) anyways.

Remember sites pushing 2-digit terabytes of dynamic html are very rare. And everything else can be trivially offloaded (e.g. to one of the isp's you advocate).

Well, I did and came back underwhelmed - as usual. Yes, you can go cheaper for a comparable spec, sometimes as much as 50 bucks. But again - that only matters when you're a small shop with 1-3 servers, looking to cut the penny.

Once your business grows out of that the hosting fees turn into a rounding error either way, and the advantages EC2 offers are nothing to sneeze at (ever looked what an solution equal to EBS costs elsewhere?).

Its no more difficult to setup a normal dedicated server than it is a EC2 box.

That's a red herring. Provisioning is automated in any reasonable sized deployment and you face the exact same problems with dedicated hardware that you have with amazon. Plus the hassles of having to babysit said hardware...

There is exactly one argument against amazon today and that is when you need hosts with lots of memory. RAM is expensive in the cloud, for obvious reasons - to a degree where stuffing your own hardware with 64G starts to look like a bargain. But the thing about this argument is: Very few applications really need that amount of RAM in a single box. In most cases the root cause for such a requirement is simply a flaw in your architecture.

"A measly hundred buck"? I didnt know we dont care about cash anymore. My whole argument is that the advantages of EC2 dont outweigh the cost benefit. When you say things like that then sure, go nuts. I have some monster cables to sell you also, superior quality for sure.

I wouldnt say text sites pushing over a TB are rare. I have a few myself doing over that ( petitionspot.com does 2+ TB a month ) - That particular site runs on a 5520 QC with 6 gigs of ram and 10tb transfer for 200$ a month )

If I was to put that same site on EC2, I would be paying significantly more, as I doubt even the large instance could handle it. 1.2ghz opterons are not exactly speed demons.

For the sake of argument, lets say the Large Instance could handle it though.

Pricing at 34 cents a hour = 252.96 a month Bandwidth costs, 2TB = 340.00 a month

So we can either pay 592.96 for a less powerful machine, Or 200, for one that has 8TB more bandwidth to play with, and has roughly twice the processing power. In case you are feeling frisky, you can upgrade the ram to 12gb for 75$ a month. Still significantly cheaper.

https://orders.gigenet.com/index.php?order=true&form=235... Heres a link to the server I mention above.

As stated in my other comments I'm mostly coming in from the angle of bigger deployments (3+ hosts) here.

That's where EC2 starts to make sense - not always, but often.

For your site, without knowing the details, I have no doubt that you made a sound decision. You should just be open to revisit that decision once it grows to a point where a few hundred bucks difference a month don't weigh as much anymore. Or rather, where the added flexibility makes that difference worthwhile. - That's all I was trying to say.

Your calculation does not consider reserved instances. Reserved instances allow you to get the cost of running a large instance down to $127/month (not including bandwidth, on a 3 year contract) or $163/month (not including bandwidth, 1 year contract). Does not change the overall picture though.

You also do not differentiate between traffic in and out ("in" is currently free fo EC2 and in any case cheaper than traffic out) - this may be significant depending on the kind of service you run (for most applications it won't be).

Similarly: if you make use of other Amazon Services (in particular S3 which you too seemed to like) this would count as external traffic from servers external to amazon but not from EC2. However, again, for most applications that would probably not change the overall picture.

But i do see one argument that can change the picture: if you use ec2 for overflow, testing and short time workloads and some other hosting (in-house?) for everything else, then this significantly complicates your infrastructure, i.e. you need take care of your in house servers and understand ec2. This may be significant enough to run even long running servers in the cloud.

I use VPS for overflow. The hourly billing lets me add in a hardcore box for a few hours when something gets popular or a celeb endorses it. Im a big fan of the cloud, I just dont think it makes economical sense to run something on it all the time. The cloud makes sense to me only when you dont run your stuff on it all the time.

That's not really a safe assumption. I work with several real-world, profitable sites whose monthly bandwidth usage range from 50Mbps to 2Gbps at 95th percentile, and they push enough custom content that CDNs are overly complex and still not priced competitively. Their respective business models would not be viable if they ran on Amazon. I just helped one of them acquire a pair of GigE uplinks, one of which they're paying $1,500/month for the other they're paying $3k/month for. Assuming they can see 800Mbps out of each link, that's $7 and $14/TB of xfe. This compared to Amazon's "measly $100" ..

Well, I'd call that a corner case - but I may stand corrected.

2Gbps is indeed a different ballpark. For most sites I have seen in that realm it would be ~90% static content (i.e. media). If your site truly needs to hit logic for each of these requests then yes, a distributed solution probably won't do there.

Anyways, I didn't mean to say that EC2 is the be-all for everything. I just wanted to place a counterpoint against that all too common, lighthearted cost-bashing which too often is not backed up by figures that make sense the way yours do.

I've been building infrastructure a long time, and for even longer, I've been fixing bad infrastructure. When I first consulted for a company using EC2, they were using it because they were just bad at planning & managing their infrastructure. Three years later it's pretty much the same story.

These kids (I'm 30 now, I can totally pull off the crotchety old man thing) drink the cloud kool-aid and think "great, we don't need to think about our infrastructure". They slap together something that almost works, but is a management nightmare. It's no different from the startups I was consulting for in the '90s, except it's a lot quicker to use a brute-force scaling approach now.

I like S3 for what I use it for: cheap tertiary archival for ~20TB worth of backups. Beyond that though, it's hard to find ways for the numbers to work for the pure cloud hosting sites. Especially for services that are more than just simple CRUD apps. None of the startups I'm currently incubating can do more than maybe some simplistic web hosting there, since they all involve "complex" infrastructure needs. (I double-quote complex, because in my world, BGP route optimization or IPv4 anycast is just a run of the mill daily problem).

You seem to have missed a pretty big feature of EC2 - servers on demand. There is no thought or overhead in spinning up a new server. 10 seconds in your console of choice and you have a new virtual server, ready to go. That, combined with instance customization, mean server instantiation can be done by a developer on her lunch break, instead of taking hours and attention away from the ops guy.

You can justify how important that is to you I suppose.

Any decent dedicated provider (eg- theplanet.com ) Will email you your server details 5 minutes after you order the box, unless you are getting some weird equipment or off the bend OS.

The bottom line is you are paying 2, sometimes 3 times as much for that convenience of "instantly" powering it on. How often do you that? How often do most users do that?

If your business is at all like most, Your need to power on a new server instantly isnt mission critical. Even if it is, You should have your main servers running OFF of ec2. By all means, use ec2 or any other VPS to handle your overflow.

I think that you are missing the value of the opposite proposition - that you are able to turn it off and stop paying. This is an obvious value to someone who needs to be able to scale sideways, but having essentially no cost for provisioning and de-provisioning allows you to do things for development, testing and operations you cannot do without great expense outside of it. I worked at a number of banks 10 years ago and we can get the same functionality now for a few bucks that we did then by buying far too many spare E3500s.

We have a decent sized EC2 architecture right now and EBS snapshotting + provisioning on demand means that we can get all of the benefits of multiply redundant hardware at a fraction of the cost.

- We never need to upgrade software in place. Software gets freshly installed on a new cluster that duplicates our existing cluster and gets a increasing portion of traffic. Failover is as simple as a proxy cutover. The freedom to do config changes or 3rd party software upgrades and be able to just swap instances in and out enables us to experiment often and safely with production changes.

- Slave gets corrupted or falls out of sync with the master? A machine suffers a degraded disk? Bad net interface? Don't fix, replace.

- We can spin up an entire test stack that precisely mirrors our prod stack to stress test with significant cash outlay. Since this cost is low, we can stress test often and test the effect of s/w and config changes on performance easily. Stress tests the H/W outlays associated with them are traditionally hugely expensive if you want to test exactly like you run.

- Significant database migrations can take place on a slave that is easily promoted, snapshotted and cloned without having to deal with any long term ping ponging between instances.

- Copies of databases can be instantly provisioned and destroyed for data analysis needs. If analysis needs peak at 5x in a week, no one has to wait. Run 5 copies and shut them all off when you're done.

These are just a few quick things that we do with regularity. And I'm sure there are lots more opportunities to exploit these capabilities than we are thinking of.

Also - in terms of cost - don't forget to compare against reserved instance pricing.

I did state in the very first post that hourly billing is the only asset. Thats why its perfect for overflow. I can spin up a double extra large and use it for a few hours when I need it, and kill it afterwards. Thats it though.

I use r1soft for redundancy and the costs for that are nominal. That sort of handles any hardware issues for me. I can get back up and running in about 10 minutes from a catastrophic failure.

All the stuff you mention below really doesnt matter to me, and I cant imagine it matters that much to many people. All these tests you mention could easily be done off site, or mirrored to run locally.

Still not worth paying 3-4x for these random "abilitys"

For things that matter- Bandwidth, Processing Power, Ram- Dedicated servers have EC2 Beat, hands down. More power, Less cost.

I can get back up and running in about 10 minutes from a catastrophic failure.

How often have you tried it?

Ive done it before when I went to a new server. Restoring the bare metal from r1soft.

It's actually harder to set up your software on an EC2 instance, but just the first time -- it demands a pretty high level of sophistication.

OTOH, do you have API keys for your Dell salesman and your co-location facilities? I'd love to see those provisioning scripts!

I moved up from a small instance to a medium instance on a server that I never thought I would need more power on, it took me less than 5 minutes to make the switch. (unattach the IP and EBS and attach to the new instance)

This is why I pay for EC2. My dedicated boxes are on a yearly contract and when it comes to move it's a couple hours to move data, IP addresses, etc.

The amount you pay for a "small" instance on EC2 is the same price as a machine you can get with gigenet that is more powerful than the "Large" instance.

Hence the major problem with ec2- Cost. Yes, you can instantly scale up to a less powerful server than you would have got in the first place from a dedicated provider.

Also, You do know you can do things like upgrade the processor and ram on a dedicated, usually within a hour also. You just send in a ticket.

What server provider makes you do a contract?

The amount you pay for a "small" instance on EC2 is the same price as a machine you can get with gigenet that is more powerful than the "Large" instance.

That is simply false. No idea what kind of beef you have with amazon but please get your facts straight.

You seem to consequently ignore that with amazon you not only pay for a piece of hardware but also for the flexibility and infrastructure that comes with it.

Yes, your cheapo host might give you a gig more RAM and a few more megahertz for your dollar - but it doesn't have equivalents to EBS, S3, CloudFront, instant provisioning and all the other goodies that make EC2 worthwhile.

It will probably also not give you means to recover within minutes when your shiney, dedicated pizza box decides to blow a fuse.

Cheapo host? Theplanet and others have significantly more servers and datacenters than amazon.

Most servers sold from either provider come with 1-2TB of bandwidth. How much does 2TB of bandwidth cost on amazon again? Right.

I already said /multiple times/ that the other amazon services are well worth it. I only said EC2 /in my mind/ is not worth it, unless its for your overflow. If you have a static need for a server- As in, a webserver that will always be on, Then you are better off getting a dedicated. The bandwidth will cost less. The server itself will cost less. You will get more processing power.. You will get more RAM.

You will not get instant provisioning. But then again, I dont randomly change the specs on my webservers now either way.

Most servers sold from either provider come with 1-2TB of bandwidth. How much does 2TB of bandwidth cost on amazon again? Right.

340 dollars. If 340 dollars matter in your equation then I think we're talking about different things here.

I only said EC2 /in my mind/ is not worth it

Actually you come across like talking ultimo, just like in your next sentence...

If you have a static need for a server- As in, a webserver that will always be on, Then you are better off getting a dedicated.

See? That sentence simply doesn't work in that general form. If you have a need for a single webserver then yes, EC2 is probably not cost effective (although it can be even there, depending on the use-case). Once you're talking about a deployment with multiple servers the picture changes. Not always in the favor of Amazon but often enough to refute such a general statement.

Looking at it again it seems like all your arguments come from the angle of renting a single server. - In that light you make some sense. But please also see that amazon starts to make a lot of sense in the mid-range and up, in many situations.

You might want to go over your figures there. Bandwidth is only 10 cents a gigabyte if you have done 150TB in transfer already. For common folk, its 17 cents a GB. == 170$ per TB

I did a rundown a few posts up where I price out the need for a fairly common php/mysql website that would cost 600$ a month on ec2 to run, yet it only costs 200$ on dedicated.

Also, Im not talking about single server setups only. Show me a scenario where its cheaper to run your entire business on EC2 than a mix of Dedicated + EC2.

Seriously, I want to know of one situation where it would cost LESS money to run my entire 24/7 business on amazon then on dedicated or a mix of dedicated and EC2.

You might want to go over your figures there. Bandwidth is only 10 cents a gigabyte if you have done 150TB in transfer already. For common folk, its 17 cents a GB. == 170$ per TB

Thanks for pointing that out, my bad. Edited it as to not collect karma for false figures. I stand by my point, be it $140 more or less.

Also, Im not talking about single server setups only. Show me a scenario where its cheaper to run your entire business on EC2 than a mix of Dedicated + EC2.

Please see my first comment in this discussion where I said exactly that. It's exactly this mix which makes the most sense in most cases.

Also, Im not talking about single server setups only. Show me a scenario where its cheaper to run your entire business on EC2 than a mix of Dedicated + EC2.

I never claimed that - but yes, even those scenarios exist.

If your business is number crunching then EC2 might be a good deal. Likewise if your business is strongly saisonal (e.g. ticket sales for sports events or such).

Moreover there are many scenarios where EC2 enables things you could not realistically afford otherwise; do you think dropbox could have realized their storage and scalability needs on conventional hardware?

While traffic might be "ridiculously" expensive on AWS, storage happens to be riciculously cheap. You don't get $.10/GB "infinitely" scalable random access storage elsewhere.

And finally, closer to your usual PHP/MySQL shop, in many companies the savings due to flexibility and availability simply outweigh the markup on the hosting bill.

You keep throwing figures in the 3 digit range. Sorry man, but your admins burn that per hour while they reinvent all the little infrastructure things that amazon already has.

Trust me, not all of amazon's customers running those 1mio-or-something instances are idiots.

First off, The "average" PHP / MySQL shop will almost never be cheaper on amazon. Are you kidding me? The average php and mysql website has one server or less. You dont have 2, or 3. So either throw out some stats and numbers or stop bullshitting people. There are enough people blindly drinking the cloud hosting koolaid without the fake facts.

Im talking 3 figures because thats what I assume most startups need for their twitter/facebook/whatever app. It only becomes even more ridiculously expensive to stay with EC2 once you get bigger. You cant just say "well, once you are big its better!" Its not. Once you get bigger, the downfalls of relying on ec2 (for anything besides overflow) are even worse. The more servers you have and the more bandwidth you do, the more cash you throw out the window.

In fact as you get bigger you would be irresponsible to your shareholders to rely on amazon and not hire your own engineer. You know why? Its cheaper to have your own engineer and manage xx servers on colo then it is to have amazon. Not to mention you will still need a server guy once you get to that size anyways. Theres a reason you dont find any sites in the top 100 running their businesses entirely off of amazon. Its not financially prudent. Its a easy way out and not the /right/ way out.

Also, we are not talking about s3, simple DB or CDS, so stop bringing that up. We are talking about EC2. You dont get to say that EC2 is awesome because S3 exists. They are separate products.

First off, The "average" PHP / MySQL shop will almost never be cheaper on amazon. Are you kidding me? The average php and mysql website has one server or less. You dont have 2, or 3.

I said multiple times that 1-3 server shops are irrelevant for this discussion. At that scale you can use anything and it simply doesn't matter either way.

Since you seem to have mild reading comprehension issues I will spell that out for you: Neither will a profitable business break a sweat over paying 500 more or less on hosting. Nor do many of the benefits of EC2 come into play at such a small scale.

So either throw out some stats and numbers or stop bullshitting people. There are enough people blindly drinking the cloud hosting koolaid without the fake facts.

I don't like your tone. I explained four possible use-cases in my last comment, I don't think I used "fake facts" anywhere. If you are immune to basic math then that's your problem, no need to get personal.

Its cheaper to have your own engineer and manage xx servers on colo then it is to have amazon.

I conclude that you simply have no idea what you're talking about, much less about the figures involved. Furthermore an EC2 instance apparently killed your pet dog at some point in the past, which now prevents you from any rational thought in that direction.

The contract isn't required, but I use it to get the best deal. And if you do the reserved instance the EC2 instances get pretty cheap.

Upgrading hardware involves more downtime, and it's not anywhere near as easy to scale back down if I need to.

If you are signing contracts to get a deal, it sounds like you are price conscious. You should see in comments above where I spec out the needs for a typical website. However, Assuming you only need 1.2ghz of single core processing power and nobody needs to visit your website, ec2 is perfect.

It only starts sucking when you actually have to serve files and do it quickly.

I haven't had a problem with speed yet, but then again I use CloudFront and don't serve files directly from the server. For me its worth the money for the flexibility and the ability to scale as needed. I might be able to save a little money if I went with blended dedicated/EC2 but honestly the extra headache and latency for cross-communication wouldn't be worth the hassle.

Reply on news.ycombinator.com