Rackspace hardware failure, all data lost. Oops... | Hacker News Reader

Rackspace hardware failure, all data lost. Oops...

13 pointsnixle12 comments

Just a friendly reminder to not put too much trust in "the cloud", it was just our development server and we are rebuilding... but not with Rackspace. The thing is that this is the only message I have received about it. It's pretty much: "yeah, we lost all your data, good luck".

===========================

Hello,

This message is a follow-up to our previous emails. Despite the very best efforts of our engineering and datacenter operations teams, unfortunately the host machine that your cloud server xxxxx resides on was not able to be recovered. At this point, there is complete data loss on the server.

If you have taken a server image, your next step will be to build a new server from the server image. If you require the same IP address, please contact support prior to deleting the failed server from your account and we will assist you in sharing the IP to the new server. If you have not taken a server image, your next step will be to build a new server from a stock image and then upload data from your local backup.

We are extremely sorry for this inconvenience. Our Billing Team is going to process a xxxx credit for the server, equal to the monthly cost.

If you have any further questions, please contact a member of our support team by responding to this ticket, visiting us in live-chat, or by calling xxxxx or xxxxx

Best Regards, The Rackspace Cloud US Toll Free: xxxxxxxxx

======================

This all depends... were you paying for a managed server that they were supposed to be backing up? If so, massive fail by rackspace. But if you were supposed to be in charge of the backups, then don't blame rackspace.

You can outsource your hardware, but you can't outsource your responsibility. Backups and redundancy are your responsibility, not your service provider's.

..unless you pay for automated backups.

You know, even so, in the end it's your responsibility to your client, even though in practice you have to draw the line at some point and just point the finger at "Amazon is down" or w/e.

Another thing of note is your service agreement. Check what responsibilities and restitution are present in the contract. If you need them to be liable for damages beyond downtime, make sure it says that in the contract. If you need them to failover, make sure it says that in the contract. If it doesn't, well... it's pretty much: "yeah, we lost all your data, good luck".

I've done hosting for over a decade; no provider with a brain is ever liable for anything more than what you've paid for.

The company I work for has cloud hosting contracts which state if the service goes down, they are liable for a certain percentage of lost revenue due to the outage. This comes with a 100% uptime guarantee, including several failovers. We pay dearly for it, but if you need something specific, you'll likely be able to negotiate a contract based on what you need. If it's not in the contract, you won't get it was my point.

Fair enough; but you're still going to limit your liability. Assume your customer spends $10K/month with you; or $120K/year. Would you ever commit to compensating them $1MM+ in the event of an outage? Of course not. It would be business suicide.

If you want that level of uptime, you build out your own place (i.e. Amazon, Facebook, Google). If you're not big enough to build your own, you live with the possibility of downtime.

True story. I took your comment two ways, and wrote my response based on the wrong interpretation. For that I apologize. Of course we pay more than the host could ever be expected to pay out. It's an insurance policy, when it comes down to it. It's also less than we would have to pay to get our own datacenter put up faster. As a stop-gap between the time our software went live and the time when our datacenter will be operational, it's worth it. Time to market was very important, moreso than the cost we pay up front. Losing marketshare forever is worse than an $8b/yr company losing a few million dollars for just one quarter. You and I are not disagreeing.

But this is all getting off topic. What I was originally addressing was the OP's comment of "sorry, we lost your stuff, oh well". If you want that peace of mind, you're going to be paying for it.

"But this is all getting off topic. What I was originally addressing was the OP's comment of "sorry, we lost your stuff, oh well". If you want that peace of mind, you're going to be paying for it."

Agreed.

Rage level depends on what you're paying for. I have a server at Rackspace, it's my job to ensure backups, not theirs. That's the service level I pay for. They offer a backup service through Cloud Files that puts your backup on a different server. About once a month (which is all I need for the little usage my server gets) I log in, click the button to do a Cloud Files backup, and I'm done. Costs me about 60 cents a month for the file storage, pretty cheap for an off-server backup. If the server holding my instance fails, it's a couple of clicks and about 10min to restore it back.

I hate it for anyone who loses data, but if you're not paying them for a backup service, it's an expensive lesson to learn.

No backups, hey? That's rough my friend and I feel bad for you. Best of luck recovering and thanks for the reminder; I'm going to make sure that my backup scripts work!!!

Reply on news.ycombinator.com