Hacker News Reader

Comment by walrus01 | Hacker News Reader

ISP here: The margins in bulk hosting services are incredibly thin, and companies have resorted to automation tools. If somebody asked me to run backend infrastructure for something like DigitalOcean or Linode, I would run away screaming. It would literally be my own personal hell. I would rather run any other sort of ISP services on the planet than a bulkhosting service where anybody with a pulse and $10 to $20/month can sign up for a VPS.

I truly feel sorry for their first and second tier customer support people. I imagine the staff churn rate is incredible.

People who work for these sorts of low-end hosting companies inevitably quit and try to work for an ISP that has more clueful customers. When you have people paying $250/month to colocate a few 1RU servers, the level of clue of the customer and amount of hassle you will get from the customer is a great deal less than a $15/month VPS customer.

View on HN

This race to the bottom has reached a point that it's harming customers. It's okay to be more expensive than the competition if you provide a better service.

Personal opinion, it's really important in the ISP/hosting world to identify what market categories are a race to the bottom, and if at all possible, refuse to participate in them.

I look at companies selling $5 to $15/month VPS services and try to figure out how many customers they need to be set up for monthly recurring services, in order to pay for reasonably reliable and redundant infrastructure, and the math just doesn't pencil out without:

a) massive oversubscription

b) near full automation of support, neglect of actual customer issues, callous indifference caused by overworked first tier support

Conversely, as a customer, you should be suspicious when some company is offering a ridiculous amount of RAM, disk space and "unlimited 1000 Mbps!" at a cheap price. You should expect that there will be basically no support, it might have two nines of uptime, you're responsible for doing all your own offsite backups, etc.

If you use such a service for anything that you would consider "production", you need to design your entire configuration of the OS and daemons/software on the VM with one thing in mind: The VM might disappear completely at any time, arbitrarily, and not come back, and any efforts to resolve a situation through customer support will be futile.

Unfortunately there is no guarantee that I receive any better support at the more expensive provider though

At a certain price point, yes there is, if you're paying $800/month for hosting services to a mid sized regional ISP with presence at major IX points. That ISP cares about its reputation, and cares about the revenue it's getting from you.

I can tell you that as a person whose job title includes "network engineer", we have a number of customers who have critical server/VM functions similar to these people who had the DigitalOcean disaster. If something goes wrong, an actual live human being with at least a moderate degree of linux+neteng clue is going to take a look at their ticket, personally address it, and go through our escalation path if needed.

Having paid substantial amounts for various services over the years, paying hundreds of dollars per month doesn't automatically make you into a priority.

There seems to be a sweet spot for company size here. Too small companies can't support you even though they really want to. Large companies are busy chasing millions in big contracts, and don't really care about your $800 per month at all.

Very good points. What I would recommend is to use a mid sized ISP in your local area where you can meet with people in person. At higher dollar figures there should be some sales person and network engineer you can meet in their local office, meet for coffee, discuss your requirements, and have something of a real business relationship with. You and your company should be personally known to them.

If you are just some semi anonymous faceless person ordering services off a credit card payment form on a website, all bets are off...

There is the same risk of being kicked off when using Amazon AWS. The rules are different, but there will be situations where you lose everything (imagine you become visible for a political reason and the landscape shifts a bit).

"you're responsible for doing all your own offsite backups"

That's going to be true no matter which cloud provider you choose.

Their ToS almost certainly include terms which allow them to kick you off and refund any monies for any reason whatsoever.

Good luck if you bought into their entire ecosystem and can't move elsewhere on a whim.

That's going to be true no matter which cloud provider you choose

My company provides services to fortune 100 companies, and we host literally petabytes of data on their behalf in Amazon S3, but we don't have offsite backups. We (and they) rely on Amazon's durability promise.

We do offer the option of replicating their data to another cloud provider, but few customers use that service -- few companies want to pay over twice the cost of storage for a backup they should never need to use when the provider promises 99.999999999% durability.

For AWS it doesn't make a lot of sense to protect against AWS itself losing data since you're paying them a premium for that. Backups in this model would be logically separated so a user/programmer error can't wipe out the only copy of your production dataset.

I don't know the data you're holding. If it is sensitive data, like customer anything, would it infact make sense not to have offsite backup?

Reasoning: Your contract with Amazon promises durability and I'm sure there's a service level agreement with penalty/liability clauses. By implementing a redundant backup, you're replicating something that you don't legally need to have, double-or-more due diligence on the offsite backup security/credentials, and in case of a failure of Amazon create a grey area with clients "Do you have the data, or do you not?"

In short, there could be a very good business reason not to do offsite backups.

Regardless of durability if you lose your customers data are you sure you will have customers paying you to keep you in business while you figure out liability?

In this case, it was not losing data, but losing access to data. The data was eventually restored. Lose customers' data could also mean losing the backup:

"We're sorry, the tape that we didn't needed to keep has been lost/zero-dayed/secondary service provider has gone bankrupt/Billy's house that we left it at got robbed." These must be disclosed to a customer immediately.

Minimising attack/liability surface is not only a technical problem, but a business one too.

It's just greed, not some wisdom. When data is lost - it's just lost. Maybe AWS will pay some compensation because of their promises, but money not always can solve problems of missing data.

Ten years ago, I had a pretty reasonable $10/mo account, that I eventually moved to a $20/mo account because I needed more resources to keep up with traffic.

I'd expect that, as more transistors have been packed blah blah blah, that such a $10/mo account would have gotten better, not worse, since then.

Support and staffing costs are not subject to Moore's law.

Neither is bandwidth.

bandwidth does drop in price, and increase in capacity, at a fairly rapid rate. Look at what an ISP might pay for a 10GbE IP transit circuit in 2008 vs what you can get a 100GbE circuit for today. But peoples' bandwidth needs and traffic also grow rapidly.

> I'd expect that, as more transistors have been packed blah blah blah, that such a $10/mo account would have gotten better, not worse, since then.

This is what Linode do - they keep you on the same payment level, but raise what you get for it.

Imho, that point was reached in hosting over 15 years ago (which is why I sold the hosting company I had back then). We’ve seen some short lived upticks periodically since then, but they all end up going back to shit as they tried to scale.

I like prgmr.com’s motto, “expect about $5/month in support”

srn from prgmr.com here. Our tagline originally came from us being a low cost service, but I like to think of it as a customer support philosophy. One meaning is we want you to be able to fix the problem yourself by giving you instructions instead of logging into your system. Another is we try give you the benefit of the doubt that it could be our problem and not assume it's yours when there's an issue.

which goes in line with “we don’t assume you’re stupid”

Reply on news.ycombinator.com