Hacker News Reader

Comment by jilX | Hacker News Reader

It’s not that I want you to do on-call, I want you to be responsible for the work you do and feel the pain when it doesn’t work so you _fix_ the root causes of incidents (or at least automate mitigations).

The biggest threat to my company was the developers not fixing the problems they created. Yes, culture problems and “feature factory”, but this is part of how you fix those. (Of course you also need management support for above, super important to look for.)

There shouldn’t be on-call burnout with the above because you have agency to improve the problems. Matched with the situation continually improving and compensation for incidents. (Realistically the target is always moving and emergent behaviours will happen but this should still lie in the range of being reasonable, given a sufficiently large team among other things).

View on HN

> There shouldn’t be on-call burnout with the above because you have agency to improve the problems.

I've never seen a situation where management felt people needed to "feel the pain" and gave true agency to the developers. Lip service agency like, "You can put the issue into the backlog." But never true agency permitting the developers to reprioritize to fix the fundamental architectural issues causing the bulk of problems (often by developers who'd left the company or advanced into management and were no longer on call).

In one of my first jobs at Fortune 20 company, they had on-call escalation procedure, longer you take to fix an issue, more and more people get called. It was something like on-call developer > team lead > whole team and manager > director > senior director > VP... Usually, after an hour, we had to get manager and whole team online. Manager would be freaking out and driving us hard to fix outage with whatever means necessary and not let it escalate further. They would make it sound like our whole team will suffer if director got on the call. Of course, they said that we can create issues and fix them and of course, there were always more important issues.

My team quickly and informally learned if you want to take care of technical debt, let the higher ups feel the pain. Even when nginx/some process died and just need to restart, we would wait and really try to let those calls escalate to director. And anytime director had to join bridge, those issues would become highest priorities and fixed permanently.

It also became a game where manager would keep asking to do simple things to fix and we would be stalling or looking at unimportant things. Pretending like slow internet connection, dropped calls, locking ourselves out from the box, etc.

And in our team we were all PS3 gamers, so we would get on GTA4, and start having fun. A few times we even yelled at each other for messing up in the game while unmuted.

These days depending on company's culture I may fix issue quickly or I may let it escalate to right level where we will get support for fixing the issue for good.

what would true agency actually look like? would the team "owning" the system get to capture the bulk of revenue or value the system produces -- so they could decide if they wanted to stack on another feature that might improve revenue by blah percent versus spend the quarter trying to stabilise the system to reduce the number of false-positive alerts or avoidable outages.

I already answered your question, but to reiterate:

If the belief is that developers should be on-call (as the OP I responded to suggested) so that they "feel the pain" and address the issues, then they actually need agency to address the issues.

If the problems are in someone else's code (a purchased system, another department), then they have no agency to address the problem beyond entering tickets.

If the problems are in code that they are responsible for, but are not permitted to change (it's too low in the priority queue), then they have no agency.

If the problems are in the structure of the system and they aren't permitted to change it (or, again, it's too low in the priority queue), then they have no agency.

They have to be permitted to restructure, recode, and reprioritize everything to have agency. Otherwise the "feel the pain" approach is just management fucking around not wanting to hire enough staff to handle the on-call rotations, but covering it up by claiming they have a "good" reason but failing at follow through.

As a real-world example of this, a place I worked at did the on-call rotation for devs, but had a problem with a message queue. The devs weren't permitted to change its configuration (tune it). They weren't permitted to purchase new hardware to distribute the load. They weren't (it was a purchased system) permitted to change the code to the message queue system. They just got calls every night saying that the queue was full and had to address it. Every single night. There were no fixes because they weren't empowered to fix it. Naturally, a lot of people left rather quickly because they got tired of management's bullshit "feel the pain but don't fix the problem" attitude.

And turnover is another issue. Even if the problem were caused by those developers (and not a component they couldn't control), turnover, which happens in every company, means that at some point the devs getting the calls won't even be the ones who made the system. They're just the poor schmucks who hired on thinking they were going to be software developers, but who got saddled with an extra job they weren't actually trained for.

If you can't remove yourself from the oncall list...

...then you already have no agency.

> feel the pain when it doesn’t work

If I have to FEEL THE PAIN when software I work on contains bugs, I need to be paid a whole of lot more than what you are offering.

Reply on news.ycombinator.com