Comment by jeletonskelly | Hacker News Reader

Comment by jeletonskelly | Hacker News Reader

Ugh, pager duty... To me it seems like it exists solely because there is a more fundamental problem in the architecture of the system. Sure, sometimes things go wrong, but if it happens so often that there needs to be an official rotation to deal with it, then it means that something is fundamentally broken. I recently passed up a good job offer because they had pager duty, and this is for a well known .com.

I think you should ask the developers in the other team how often they get called during their rotation. You should also ask how much of a priority it is within their work scope to eliminate the issues that are causing the processes to fail.

I used to work for a small company that had nightly batch processing jobs on stock data from that trading day. If any one of those processes failed, then someone had to log in and fix it or the company wouldn't have a product for the next day. During the day we had other things to work on, things the business wanted and there was little importance given to fixing the brittle (broken) data processing. Management saw it as working software. They weren't the ones logging in at 3am for two hours to keep the business rolling the next day. That had a big effect on me. I felt like they didn't care about building good software, testing the software, and giving the developers peace of mind that what was in production was well tested and signed off. This is what ultimately led me to leaving that company and joining one which had solid processes: development -> staging -> qa -> production. Because of that process we haven't had a single outage in 3 years. I can go home at night and think about the software I'm currently building, not worrying if I'm going to get an email alert late at night because no one cares about fixing our broken software/processes.

In conclusion, take heed.

> "Sure, sometimes things go wrong, but if it happens so often that there needs to be an official rotation to deal with it, then it means that something is fundamentally broken."

People who say things like this are usually the source of problems for the people who carry pagers - this is why in DevOps world, developers now join the club. ;)

The notion that you wouldn't rotate people if you don't alert very often is not only silly, it's why I left my last two jobs. No "rotation" meant Justizin is fucking always on-call. :D

There's a big difference having someone who is responsible for the operation of the infrastructure being on pager duty and the developers being on pager duty. In most development shops there would be no point in having one random developer being on pager duty. What if something breaks that he's never touched? Usually devops is first made aware of the issue, determines the cause, and, if it's a software issue, should then reach out to the developers who are responsible for that part of the code. A pager rotation among developers means there's something fundamentally wrong. If devops is constantly being paged because of shitty software, they should be the first to recommend that the QA -> release process be evaluated because it's clearly broken.

I'm very confused by your reply.. are you trying to say you have three separate groups? ops, devops, and developers? With devops existing to communicate between ops and developers?

> if it happens so often that there needs to be an official rotation to deal with it

Frequency doesn't matter. Even if issues are rare, for many companies it's terribly important that the issues that do arise get fixed promptly no matter what time they occur.

Your company should very definitely care about the frequency of off-hours support issues (your previous employer apparently did not) and work to reduce them if they're anything but extremely rare, but somebody should still be on-call.

Depending on the size of your team, that somebody should be in ops. That said, if you're a smaller team of only developers, you're right, someone should always be on-call.

Totally agree. Sometimes pager duty exists because you have broken processes that the company won't invest in fixing. That's why I was woken up every night for 6 months - A crawler trying to download the whole internet would run out of memory and trigger our alert. We weren't making any money on the project so we weren't allowed to invest any time to fix it, but we could not turn off the alert either. We had many processes like this one, though not all of them failed every night.

Reply on news.ycombinator.com