Hacker News Reader

Comment by vitovito | Hacker News Reader

I have to assume all of the other comments in this thread are from small shops that have never supported a live product.

We run a multi-hundred person team here for a live, 24/7 product, and as many as half of our developers have been scheduled as "on-call programmers," which we call our Live team. Their sole responsibility is the live, deployed product and customer-impacting issues.

They do no bug fixes outside of that. They do no feature development outside of that. There is an entire other team dedicated to those things, and like 37s, that team gets rotated through.

We also have QA dedicated to the live product, Operations dedicated to the live product, etc., etc., all separate from new feature development, because an immediate, customer-facing issue requires different prioritization than feature development.

View on HN

I've supported (what I would expect to be) an equivalently large deployment. If you truly have half of a multi-hundred person development team scheduled simply to respond to emergency on-call events, you very, very likely have fundamental issues in your development standards and processes leading to those events.

That's simply a tremendous percentage of your staff dedicated to putting out fires.

Well, some of them are artists and designers, too, and this isn't just a web site, it's a desktop product and an online service, and the proportion changes depending on where features are in development and what sort of load we're seeing on customer-facing issues, but, yes, there have been occasions where half of our web and infrastructure staff have been doing "live" development and support.

And that's the thing: they're not "emergency on-call" events. They're simply "customer-facing issues." With a 24/7 product and 1.7M subscribers, things come up. They're not "fires." They're "live" issues. They're always there.

The 37s post is not about emergency staff, even if they're using those types of words. It's about having dedicated personnel to handle technical issues arising from a customer support ticket, so the "new feature" programmers don't have to get pulled away unless they have the only knowledge of that particular system (which doesn't happen too often here any more).

But root-cause analysis of the issues that arise is still important, right? Tracing customer-impacting issues back to the decisions that might have caused them? Perhaps not even particular bugs or parts of the product, but architectural decisions as a whole? Or even organizational processes? You can throw money at live support until the end of time, but the only way to reduce that cost is by addressing problems at the source, be it code or process, or something else... (right?)

Sure, and the "live" team is made up of people from the regular "new feature" team, rotated in and out so they all understand the impact their code has.

But, in a product that gets used by lots of real users, shit happens. You're never going to get everything right the first time.

Reply on news.ycombinator.com