Ask HN: Who else is working/on call over Christmas? | Hacker News Reader

Ask HN: Who else is working/on call over Christmas?

162 pointsdeathgripsss98 comments

I volunteered this year to be the on call over Christmas and currently I'm dealing with a generator fire. Merry Christmas.

Who else is working through the 25th this year?

View on HN

Best of luck with the generator.

I'm on a hotline, volunteering, for people who might be feeling extra mental stress due to the holidays and might do something they regret. I decided not to risk the travel to see family with the storms and I figured I might as well try to help others who might be dealing with some of the same tough emotions that often plague me this time of year.

you're doing something good. here's a virtual hug.

You are doing a ton of good this Christmas. Thank you for your efforts here and there.

The world always need people like you. Thank you.

Thank you! Have a Merry Christmas.

Good on you :)

I'm on call. I volunteered. I wasn't scheduled, but the guy who was scheduled, it is his first year with the company, and this is his first job out of college. He was visiting home, and I wanted to make sure he got a chance to go back and have a good time with his family without having to worry about on-call junk.

You know, when people go out on a limb for me to do things like that, I never forget them. Cheers

You are a cool guy!

It sounds like your on-call should be structured so that people aren't allowed to be with their family on important holidays. My company does this as well. I find it distasteful and exploitative. Of course, the response is always "just find someone to cover!" You're a good person for doing that. You shouldn't have to. Your family wont be around forever and no board of directors is worth a single minute with your family.

It's a lot easier to cover on call if your company is fully remote, or at least the on call staff rotation is remote. Employees tend to resent on call a lot less if they're home with their families.

This is so true, and well said.

I lost a few family members this year and it made me realize all the bs baggage that comes with employment.

Needless to say, I don’t do on call anymore.

On call, but this is the best time of the year to be on call: none of my coworkers are around to break anything. A gut estimate of my pager load is that 80 percent of pages are caused by bad changes pushed by humans, and another 10 percent are caused by changes pushed by humans exercising latent errors in the system, and only the remaining 10 percent random shit breaking.

The main downside is that I also don't have as many people to escalate / help when an outage does occur over the holidays. But that's not as urgent now that say, generator fires are Amazon's problem not mine.

This time of year, you do get the pages for things that were always broken, but nobody noticed before, because they only show up when the system has been running without changes for more than two weeks.

This actually happened to us last week in fact.

No deployments revealed how a legacy background processor started losing connections to the message queue and gets stuck in a state where it never reconnects.

Deployments always cycled the pods before the issue manifested.

This is something a (now former) colleague of mine pointed out: that the kubernetes descheduler can enforce a maximum lifetime[0] that sort of forces continual reboots. So if your system cannot tolerate running for a long time continously, this is one method to gracefully restart long running pods.

[0]: https://github.com/kubernetes-sigs/descheduler#podlifetime

“Generator fires are highly correlated with Christmas weather” is the best sales pitch I have heard for cloud services.

Own a little SaaS. Gave everyone the week off between Christmas and New Years. Have been happy to do this 5/6 years we have been in business. Love giving it to my team for all the hard work they do all year.

Machine wise k8s is rock solid and I never need to touch it.

Client wise, will be a few support tickets…

> Gave everyone the week off between Christmas and New Years.

Do you mean additional to their PTO? That's nice, many people feel obliged to take it off for family even if they'd rather do something else with a whole week off and see family on weekends and such.

Yes, kind of a forced vacation week. I think people come back in Jan feeling less burned out… vs having to sneak some random work days in on Dec 28th or whatever.

That’s a nice gesture; hopefully you can keep it up!

I once worked at a place that tried to always give close the office two extra days during that span. As long as Christmas wasn’t a Wednesday, they always made it such that you had an extra long weekend twice in a row, and announced it in early November so people could plan well enough in advance.

They also, for planning purposes, assumed that there were only 11 months in a year. That is, each person was only expected to do one full month of work total in Nov and Dec. it gave people time to catch up, work on tech debt, learn something, experiment with something, shop, take vacation, etc. I’d say that more often than not it was overall more productive.

It's mildly common in the US moreso in the blue collar side of things, but I've noticed some tech companies that do it too. Mine does.

It's not like anyone gets anything done.

Very nice of you. And I second k8s's robustness. Have a little one person startup with 100k users. Went on 5 weeks vacation in October and November and never had to worry about a thing. Everything kept going and just had to answer a few support mails here and there.

Ever since I’ve been in charge of my first team, this has been something I believed strongly in. One of the highlights of this holiday season for me was finding out that the new lead who took over one of my teams is doing the same thing.

Out of curiosity, what were the circumstances that made this not workable for the remaining 1 of the 6 years?

Ah, I was like 2 weeks post my 3rd baby with the others still pretty small. No way I could leave Mom alone!

But awesome enough someone from the support team stepped up and took it for the team so everyone else got the week off! So it’s cool to have a cool team.

And hey, we don’t have to rely on anyone else to work over Xmas.

Good on you for giving folks extra time off.

Very grateful for my clusters right now

You hiring?

Always

We're a remote company so we have employees that celebrate different holidays.

For example, our Chinese employees cover Christmas and we cover them during the Lunar New Year.

I'm not working but oncall. Fingers crossed. I always volunteer (I did a few years in a row) as usually it's quiet (no deployments, people understand it's a holiday) and it's a small price to pay to not feel bad for always trying to sneak out of being oncall for new year's eve :). .

Best of luck OP and merry Christmas too!

Me! First Christmas at my new (fully remote) job on an SRE team and I'm on call through most of this coming week. Not complaining though, I actually asked for this stretch, due to awkward timing with other commitments during other weeks. Fortunately there are other members of the team I can reach if I need them, and family live nearby. I'm just thankful to be gainfully employed.

It's pretty quiet at the moment and I'm actually kind of jealous that I'm not facing the kind of excitement you are... I guess I should be careful what I wish for lol.

Merry Christmas all!

Sorry to hear about your generator fire. Power stuff is usually a major drag because most of us on the IT side can’t do anything about it.

I’m on-call ops support for a public safety answering point (a “911 call-center”) this weekend. It’s a regular call rotation and my number was up for this one. It’s usually quiet— except when it isn’t.

I’m gonna go on-site to a Customer’s on-prem server room tomorrow morning and swap out a bad drive in a SAN. I’m pretty happy because I know there’s no chance of a “Hey, while you’re here-” event! Empty offices mean no unexpected questions!

Firefighter/EMT here. Always on call when I'm home. Hoping we don't get a car crash tonight because it's damn cold out there. Turnout gear helps but not enough.

I own a one-person wholesale bakery. My boss is a jerk.

Best to everyone.

Currently putting the finishing touches on a proposal that is, for some unfathomable reason, due the 28th of December.

This is my wife, who has to turn in a research paper on 31st.

On the flip side, I get to spend Christmas helping to make sure that related academic code doesn't succumb to any stereotypes ... :-)

I'm on call every day, but so is the CEO. The startup life :) I am not working this weekend, but I am the next Sunday (the 1st) so that one of my crew can have the day off. I don't celebrate the holidays, so I go out of my way to accommodate those who do, since my schedule is wide open. It's only right, and my crew really appreciates it.

Serious question - do you have stake in the company the same way your CEO does?

You are incredible. This is a great example of how to be a decent human. Thank you for sharing this and for creating a place for people to talk about it. I volunteered today to identify, move and secure the community boats on the harbor used by elderly, teens and children that were at risk from floating away with the California King tide rising. My family is a few states away and I couldn’t afford to go. I thought I would feel better just doing some things for other people. I help out 3 women in their 90’s with their technology, will stop by with food for all 3 tomorrow. I want to make the best of it. Thank you for setting a great example. You are an angel.

I’m not on call, but I’m really happy that the guys from home alarm provider are on call. We had a maltunction and they promptly reacted. Cool - and the merry evening could go on. Merry christmas.

I am doing a python freelance job. It sucks.

I’m on call this week (usual rotation, just happened to coincide). Full change blackout means less humans breaking things, and I get to save my annual leave during the mandated company shutdown + get alternate PTO for the public holidays. So far the phone has only rang twice all week. Merry Christmas!

This is me. And fingers crossed it’ll be an uneventful few days for us both :)

Volunteered to be on call -- here's to hope I won't have to press 1 to ack, 3 to nack, 7 to STFU.

I'm not this year, but I have been on call during Christmas and other holidays before. It is usually pretty quiet, both because there aren't a lot of changes with less people working, and at least for me, there are less customers using the product. But it can still be stressful knowing your celebrations or time with family can be interrupted by an alert. And sometimes you can get unusual alerts, like alerts caused by unusually low traffic, or a bug that only impacts one user, and that user happens to be the only one using a service at midnight on Christmas eve (is that you santa?).

Did it last year. I was divorced and had no reason to go anywhere during the holidays. It was my first time ever staying in the Bay Area during Christmas.

This year - I don’t have a job because I quit and went traveling and moved to Manhattan. Guess my time spending home at work in the Bay Area wasn’t great. Only thing I’m doing today is going to the gym and taking a yoga class and doing legs if I can get some time at the squat rack. Surprisingly busy yesterday (like busier than I’ve seen it almost any other time!) - we’ll see if busy today too.

I was surprised to find that my startup's contract dev kept working on the 23rd and even 24th. If he's pinging me with questions and sending me new versions to test, I'm going to answer and test stuff out! It's not mission-critical though (it's an manifest v3 transition, which Google has given a reprieve on), so we'll take a few days before getting back to it.

The good thing about testing browser extensions is that you can do much of it while just surfing the web! Cheers to everyone else who are clocking in during the holidays!

On-call here, but it's mostly because it arrives on a weekend, all I'm required to do is Acknowledge Alerts that are usually false positives, unless some big service disruption occurs

if the alerts are usually false positives you need new alerts!

Yes, but not so simple. Especially when the alerts are caused by 2 systems down the line and aren't owned by our immediate Team. That and the fact that we're migrating to a better thought out architecture which has been 2 years in the making. Slow pace at big corps..

you definitely shouldn’t be on call for alerts managed by another team. a lot of smells in the observability story here, i’d be pushing for an overhaul

^ this

Technically not because my boss insisted since I started in September I can wait until January. But my teammate left me as a shadow secondary (for learning purposes mostly for daytime pages), so if my team gets paged for anything I'll also get the page. I don't entirely mind at least to be aware of things going on. Fortunately our services are generally fairly quiet compared to my previous jobs'.

Hope you all have a quiet few days!

I'm on call for one day, since my team split the holidays to ~1 day per person rather than give someone the whole rotation. Not a bad arrangement imo.

In my startup experience, the pager never really ends. I’ve got one eye on my alerts inbox, and one eye on John McLane complaining about California. Still, I would rather be doing what I want, the way I want it rather than spending my days paying lip service to who ever might be within earshot. To those that choose the hard path rather than the easy lucrative one, I salute you.

Yup. on-call, and working through the holidays. Thanks to the frustrating miscalculations of corporate upper-management. And, since I am on h1b as well, cant really quit my job either without changing my personal life drastically.

Need to spend another 3 months of leetcoding, and system design to "prep" for another set of interviews. Not looking forward for the 12 hr coding days.

I'm on call this past week and next week!

My on-call is a bit odd because it's both as OSS maintainer deciding whether I need to coordinate a hotfix release, as well as certain functionality within a managed service (PaaS) enabled by the OSS project. Only the latter thing can actually page me, and I don't anticipate being paged. Fingers crossed!

christmas is the busiest time for one of my customers, and just last week one of the data servers had hardware issues that ended up in everything but the disks being replaced because the hosting provider could not find which component was actually broken. i am busy rebuilding raid since, but already 2 out of 3 raid devices restored. it was touch and go for a while, with a risk of loosing the whole server (because raid kept claiming that disks were faulty when they really weren't) i am already rethinking the architecture to make it more robust and am trying to convince the customer to invest in new servers for that.

i am the only one in my role, so i'll be monitoring the rebuild process over the holidays. that said, i don't celebrate christmas anyways, so in a team i'd be the first to volunteer to be on call at this time to allow others to relax, while i'd be taking time off on my own holidays.

I'm on call for the Netflix CDN, but luckily no one touches anything, so the number of incidents is super low.

I don’t understand the ones that are volunteering being oncall. I don’t think you are making a favor to your colleagues but to the company you work for, and that usually means: the company is being cheap regarding the on call topic. I have zero empathy for companies not handling on call the right way.

what is the right way?

If a company needs 24/7 coverage they should pay to have that coverage.

who says they aren't paying? you still need people to volunteer working during holidays

if they are paying, it's not volunteering.

Keep increasing the $ - you'll eventually hit a tipping point where people will fight over it ;)

it's volunteering to give up you legal holidays for work. in some countries, working on legal holidays automatically qualifies for higher pay, but that aside, i think it is fair to ask for volunteers first in case there are people who don't have family to see over the holidays or are not even celebrating. in fact i get pretty annoyed to not get to work on days that i don't celebrate. i'd rather have different days off that are more meaningful to me. of course, if there are no volunteers of this kind, then other incentives are a good idea, as is a rotation maybe

Managing the dispatch of our delivery drivers and monitoring our infra over the holidays.

Wish there was some way to bond with others that are on-call while being on-call. It’s lonely with the rest of the company being away. I guess this thread is a close as we get.

Best of luck to you and anyone else with generator fires and Happy Holidays to all!

I'm on call during the Christmas week 2nd time in a row in my team. Nothing earth shattering so far. There's one issue that could be resolved earlier (by the previous oncall) so it wouldn't cause more delays in other parts...

Merry Christmas to all those are on call now.

I'm never and always on all, since I don't actually have official on-call duty, but if our system breaks, I'm pretty high up on the list of numbers people will start to call for help.

Luckily thus far the worst was a 06:00 wake up on one Saturday morning.

Merry Christmas & good luck. I was scheduled to be on call but a lifesaver of a coworker covered for me. Turned out my flights were canceled and I’ll be missing the holiday with my family after all. No active incidents, thankfully.

I volunteered to be on call while off work both the past week and the next week, since I know it's going to be so quiet. 8 pages over the past week, only one of which was after hours and not caused by people messing with things

I am a volunteer at a research lab right now in another state over my winter break. Had the opportunity to take a break, but I don't know anyone here and won't be here for a long time so thought what the heck

Meeee (OnCall). But my team is small/I don't really have to do anything (fingers crossed). I do dislike that my phone volume is on, when normally it's off, so every notification is making noise

Press 4 to acknowledge.

You can make an exception for the pager duty providers number. That will it will always make a sound no matter what.

Thanks for the tip

On call, but not too worried because most of our breakages are usage based and not much usage on Christmas. I've been paged on Christmas morning before though, so I know how that feels.

I’m on call, been paged 3 times already for a high CPU on some VMs

I love carrying the pager during holidays. Nobody breaks anything ;-)

I usually volunteer, whether or not I actually end up working varies.

Fingers crossed for this year. I'm hopeful, we've done well to improve reliability and coach the junior teams

I am, but from experience it will be uneventful. We didn’t deploy any significant code changes all last week. The few we did do are all minor and very very low risk.

On call this week. I don't celebrate Christmas, so it's become tradition to work end of year while other colleagues take over during Eid.

Just had a pointless alarm go off that resolved after a few minutes. According to the Runbook there isn’t even any action that can be taken!

nonactionable alert? thats a paddlin

Working on my own startup so putting my all into it.

I’m on call too. As a solo founder of a small SaaS.

I’m curious if there’s a community like a Slack channel for sharing this kind of experiences?

I am always on-call. Pages that fall through, end up paging me. Happy Christmas, New Years everybody and stay safe!

I’m on call for a payment provider and have dealt with emergencies basically since the very beginning

I'm on call 24/7. One of the downsides of a small company.

I go months between needing to do anything though.

Finishing my PhD ... Have a job lined up for February and I've got so much writing left

Weird flex but ok

Good luck, I hope it calmed down a bit OP. Merry Christmas from fellow on-call to you, HN

I am marking exams. My deadline to enter the final grades is December 27th.

I'm sure you'll get through this!

on call, in case someone has a major cyber incident. so far, so quiet

I'm sysadmining because there's noone else who can.

Running a decentralised network. Some nodes not behaving well.

Deadline confusion :)

Reply on news.ycombinator.com