Comment by Hippocrates | Hacker News Reader

Comment by Hippocrates | Hacker News Reader

My company used Rackspace back in 2013. It was my first gig where I was "managing servers" and "deploying to production". The rest of the team was as inexperienced as I was.

We paid extra for the support services that entitled us to unlimited 24/7 human support. Whenever I was stuck on an issue with deploying, scaling, crashing etc. my boss would say "try calling support." Initially you'd get a first-line operator but I learned quickly to "escalate to a sysadmin".

The sysadmin was eager to help and would log into your VPS and do anything you'd ask, plus advise and implement on stuff they thought you'd need! Debug your running app and code? Sure. Fix broken packages or get something to compile? Of course. Custom scaling and deployment scripts in bash? Gladly. Some reps were more helpful than others and I got to know them on a first-name basis.

In hindsight it seems totally absurd to have a phone support running bash on our VPS but it was super helpful at the time. I learned a lot and they saved my bacon a few times. I can only imagine the liability issues.

We had Rackspace and used them for our managed servers at a prior job. We had a machine with 18-24 cores and something like 768GB of RAM that handled all of our VMs, including our SAS server. It had a bunch of native SSDs and it was connected to a 20TB JBOD of SSDs and a 100TB JBOD of slower spinning disks. They recommended this setup to my predecessor as it was WAY less money than a similar SAN configuration.

So. One day the two JBOD enclosures just start failing. We lose EVERYTHING. Ok, that sucks, but Rackspace handles all of our backups for us through our SLA. Let's get that restored while they figure out the JBOD issue.

Two things come out of this:

1. Their 'fanatical support' were 'backing us up' but they never actually tested the solution and...we lost everything. There were no backups.

2. They sold us the JBODs less than 18 months prior but they no longer supported them and thus we'd have to invest in a SAN--which was a crazy investment more.

---

We had already been planning a transfer to a new local data center over the course of a year's time. We called up the new company and told them we needed to move over to their shit in 7 days. They had us up in 2.

I'll never consider Rackspace for anything.

About which year did that happen? (The data lots etc)

We were using Rackspace too, at a former workplace, for a while (no crash though).

I'm guessing late 2015. We went to the interim provider while they did a slow planned migration to 100% GCP which was completed after I left in late 2018.

Yikes. A backup solution that was sold, but not configured (at all! not to mentioned non-tested backups), was a very frequent occurrence whenever I inherited clients (in the 2000-2010s, before AWS took over the world, or simple VM snapshots at most other hosts).

(I'm not defending anyone here, I think it's how horrible many hosting companies where was back then, or still are, whenever they have sales teams that only aim to up-sell stuff and do not bother getting it properly implemented)

Yep. I spent similar years at an MSP, and there were multiple occasions where standard backups were not provisione, despite the client paying for them.

I’ve had a similar pleasure in the early 2000s. It may seem absurd, but from a business perspective it makes sense: so many support questions fall into a gray area of “this is really not our business because the customer doesn’t know X, but without solving it they’ll be unhappy”. It makes sense to just say “fuck it, we’ll help them with anything, as long as they’re paying”.

It’s a similar approach as we take at my current employer. If you’re using our database products, even if you’re stuck doing basic python stuff, don’t know how docker works, or an absolute noob at AWS, we’ll help you. The time we spend is cheaper than losing business, and enterprises love it.

We do something similar on the non-tech side. We offer a saas product but most of our questions are more around how the person should be doing their job rather than how to use our software so we have a small team of experts that can answer those questions, they can also help with content creating on the marketing side too which is handy. We constantly hear from our customers that this level of service is one of the key things they really value from us.

+1 Build the customer relationship.

I probably know some of the guys who helped you out and there's a couple great ones still there. The legal team came up with the phrase "reasonable endeavor" (they didn't like best effort), and they wrote a disclaimer form they'd have customers fill out that what they were about to was beyond official support. As long as you signed that, support was usually pretty happy to "give it the old college try" as one told me. And I'll let you in on another little secret - a lot of those sysadmins were learning along with you. One of my very early projects was building out a precision time protocol service for a customer who asked for it - which ended up being a good learning experience for me.

I was with a company back in 2007-08 who was using Rackspace. I'll never forget the day I was sitting at a command prompt looking at htop trying to figure out what was straining the server so much...when I got a message in the terminal from Rackspace telling me exactly what was happening and what we needed to change to fix it.

Support person just chatting with me right there in the terminal, without me even having to call for help. It was incredible and I'm really sad to see them struggling now. That turned me into a fan.

When they picked up Slicehost, I was initially really happy about it but the benefits never really seemed to materialize. Then Digital Ocean appeared as essentially "the new Slicehost" not too much later and progress continued from there.

Just never adapted as a viable cloud option. For a while there, before GCP and Azure were prominent a lot of conversations were "Amazon or Rackspace".

That seems pretty sketchy from a data access perspective.

At the time, I thought it was pretty cool.

IIRC part of the pitch and the cost was that they were literally supplementing your IT department with the same types of agreements you'd have for an outsourced IT provider.

You will be surprised how much digitalocean has access in your cloud infra. We had a support ticket, the support person could see all our pods, namespace and kubernetes objects, disk mounts etc.

How is that a surprise? They quite literally have physical access to your VPS (even if not easily).

All the things you mentioned are simple metadata available via query.

If things went really sideways on one of my servers, and I could not manage to get back in (even with their 3-4 alternative methods), I fully expect DigitalOcean to be able to log in and restore access, and charge me for the privilege.

Question is how would you know the support engineer who has access to your stuff and can log you in is not a bad actor.

Seems like it's the default that employees have privileged access like this unless it is explicitly designed out of the system. It's hard to do that even in domains much simpler than a cloud.

Managed services, managed access

Unsolicited direct inspection of customer content is what’s surprising.

I know we don’t have homeomorphic encryption, but typically policies (requiring explicit user request / Consent) would be in place even if they can’t be sufficiently enforced with current technology.

The whole thing reminds me of that Azure LinkedIn thing (where someone was contacted on LinkedIn by a Microsoft PM based on their azure activity).

It was pretty normal in that space. Good MSPs proactively monitored their customers servers for issues. I certainly would have sent a wall message if I saw the customer was on the machine.

Understandably so! There is a relationship that I find difficult to explain here.

For a managed service provider, we were there mainly for the operating system. If told about particular services, those too.

To tend to that, we had to maintain more access than what common convention today tells us should be comfortable.

It's really sold as "an extension (or replacement) for your IT department"

We would do everything we could to avoid going even remotely near the data, but it does depend on a bit of 'trust us'

There were varying degrees to it, too. Some customers would get notified if we even pulled their IP addresses, others only if we logged in, some if we escalated privileges at all.

Like the peer comment from TheHappyOddish mentions, sending a wall message to folks logged in was a sort of common courtesy; chances are we're both in here for the same reason.

One thing to remember is that norms changed over time. In the 90s and 2000s it was pretty widely accepted that a sysadmin could access customer data pretty easily and that trust was basically factored into the relationship. Over time and various scandals our expectations have changed but there's definitely a generational divide here.

Not to detract from the paradigm shift, it is very true... but this is also more of an 'upsell' now.

Providers generally don't want this access, but they'll accept it - and your money

It's a compliance requirement in certain fields, too, so someone asking for it is a reliable way to tell that they're probably less cost-discriminatory.

Fair point!

I should have realized this was relevant and included it... that's been eating a fair bit of our time lately.

Have they changed? I assume Amazon, Google and Microsoft all could enter any "instance" (VPS) they choose or simply clone the disk.

I used to be one of those phone answering sysadmins for a specific customer revenue. I think that job gave me phone anxiety. Every single different call was some new destroyed system by someone who had no idea what they were doing, hacked systems, unplanned migrations, whiteboarding-in-your-brain architecture discussions, etc. The other webhosts I worked at would never put an admin on the phone with a customer directly unless they were seriously enterprise, like a portion of the datacenter.

One time the CTO of a major company called me directly and I had to migrate a SQL database that was the backend of their public website while having him breathe down my neck. Lots of calls like that.

We gave great support but at cost. That was tiring and we did it in shifts. I hated the days I had to do that.

FWIW for people with different support experiences, support was based on revenue. The different tiers (smb/enterprise, etc) assigned you more senior admins, etc. It wasn't a massive pool of admins or anything.

I quit eventually because I hated answering the phones. Great company while I was there, though. I recommended it to everyone who could deal with the phone calls and San Antonio.

Thank you for your service.

I'm pretty often surprised by how good AWS premium support is. Every time I've asked them a question they seem to have gone to quite the lengths to gather information even when the issue isn't really theirs.

Reply on news.ycombinator.com