Comment by mgkimsal | Hacker News Reader

Comment by mgkimsal | Hacker News Reader

Parent Full thread

Initial experiences with Rackspace in late 1999 and early 2000. I was amazed that every time I called 'support' I got a real live person. I called at 3am once on a Saturday, got a live person answering, and was connected with a competent support engineer in about 90 seconds. I was impressed.

The server(s) we had cost a lot, but knowing you could connect with someone competent 24/7 was amazing.

Then... a few months later... we had to use it, and it was good, but they were about 2 minutes away from wiping our main drive. We'd requested a new beefier server and requested either the drive be moved or everything copied over to the new server.

Got a message new server was up - great. Logged in - nothing. Connect to old server? Nope - offline/disconnected. Ugh. Calling in got me a live person, and I stressed the urgency. We did get connected with someone, and they did find the right person, who was about to wipe the previous server. Crisis averted, but... immediately I start asking why the support ticket showed all requests as 'done' when... they weren't.

We stayed a while longer, but it felt like they were growing too quickly even then to keep the 'fanatical support' model that initially attracted us. FWIW, I had a couple of dedicated servers at a small place in NH, and they were... $70-$80/month on average. The rackspace servers typically started at $500/month. I think we were paying about $1800/month for the setup in question at the time of the crisis. My belief in 'you get what you pay for' was impacted that day.

I had another client years later (2010? 2012?) that used them and it felt like a completely different beast, far more generic and commodity like every other competitor, and I've never since looked at them again.

Similar experiences, they were amazing for the first couple years I was on a team that worked with them. They really provided amazing support and took a lot of the burden of managing infrastructure off our smaller teams. Then it changed, the writing was on the wall as public crowds grew but one of my last experience was of them wiping out one of my databases doing what should have been a non-destructive drive expansion. The kicker was they restored from backup, a backup from the previous evening and didn't bother to inform us. We found out when customers started calling asking where their previous days data was. Accidents happen but the whole cover up was unforgivable.

Question for HN crowd: What would a small startup even do in this case?

- Talk to a lawyer?

- Wait for customers to contact support, or send out an email?

- Explain that a third party cloud service was at fault, or just give some general apology?

Any company, even a startup, should have procedures in place for what to do when data loss occurs. I would hope that any service I’m a paying customer of would proactively reach out to customers in the event of data loss. Blaming the cloud service provider isn’t enough because backups are ultimately your responsibility. But doing the “right” think is pretty tough if your cloud provider is not being honest with you.

Never trust ANYONE, even yourself.

You should have "oh shit" backups with another provider entirely, and in-house offline.

"TNOEY" coined.

Rackspace hosted Techcrunch after we we loved to death on the old host and every single person there that we interacted with what not just competent, but went out of their way to make sure we were live and thriving.

During one weekend the staff there split our wp instance between a separate web and db host + varnish without us even noticing because of increased traffic

The additional hosting cost (which they waived for us in exchange for a sponsorship slot) were more than worth it when you consider what consulting costs were/are. I realize this might be a biased experience because of the high profile of TC - but i referred many people there during that time and only had awesome ++ feedback which was similar to our own experience

I knew something was up when a lot of those smart people who were on the email chains suddenly responded with mailbox not found because they had left.

It made sense then that the model of having competent people managing servers for small-medium businesses doesn't really work since those same people can earn much more while you're only ever going to get ~hundreds to single digit thousands from customers for that level of service. The economics of hosting + what is consulting, and to clients who pay bills on credit cards, just doesn't add up

It made even more sense now after reading this article and knowing that the techs were pushed out by MBA's and finance people.

I had very mixed experiences as a customer in ~2009 - it was definitely possible to get through to someone very competent, but it was pretty difficult. We were trying to use their "cloud" VPS offerings (I believe it was openstack-based, and I liked the idea of supporting something more open than AWS), which were extremely buggy (for example, we had to default-retry every api operation multiple times in scripts because pretty much every type of call failed 40% of the time.) I generally had better success getting information about outages by reaching out employees I knew on freenode than I did through official support channels.

Exactly my experience with Rackspace.

Reply on news.ycombinator.com