Hacker News Reader

Comment by salt-thrower | Hacker News Reader

I did web dev work for an airline and we couldn't even release new updates to the loyalty benefits website during a weekday (even though there was no connection to operations or flights). The fact that this company would reboot a critical comms server at 3 pm on a Friday is indeed bizarre.

View on HN

I believe not deploying on Friday is an anti-pattern for most software teams.

What industries have you seen teams deploy on a Friday afternoon?

It happens. My company has done a couple dozen production rollouts today, including minutes ago.

Uh I'd imagine any team that mainly has their user base using the product on weekdays would deploy Friday evening so if something goes wrong they have the weekend to fix it.

Only if the team wants to work weekends?

If something goes down on Saturday there’s a good chance I’m out of cell service

It is interesting to see this bifurcation in action.

Hey software engineers - you know that working weekends is unusual, right? Like... please tell me someone has at least negotiated your right to only work 5 days a week so if you're working a weekend you get Monday and Tuesday off.

If not... Has anyone told y'all you should unionize?

I mean, you make it sound like releases fail often. For our team I think we have only seen one failure in the last two years needing a fix after hours (going into the weekend)?

The other option is to fail during the workweek. If the software is directly generating money for the users and the developers are judged on what actual business value they provide then failure during business hours is worse.

Did you know that when you use your 2 days off to go shopping, someone has to work at the shops?

Yes. That's why people work staggered schedules. Folks who work weekends at shops usually (and should) have two other days off in the week.

People literally fought, bled, and died for our right to not do capitalism 24/7/365, because with no back-pressure that's what the efficiency engine will demand of us.

And part of that is software engineers doing deployments after-hours for their customers. If your company or country is any good, they'll give you double time off during normal hours.

Many people work shifts, it’s quite normal.

What seems to be unique in American companies is you’re expected to work shifts or nights but also work m-f 9-5 at the same time, which is really weird.

Same with of call, if you get called at 3am on an on/call rota for a specialised third line escalation many companies still expect you to work the next day, and they might not even pay you for being on call?!

Such a smug privileged answer given in a world where the industry is shrinking its workforce and pretty much all other industries have flatlined for years(meaning we are still better off despite this realities).

Wanting to have a weekend is smug and privileged now?

People fought hard for the 40 hour work week and 2 day weekend.

You think that should be given up?

We do for clients who have 24/7 users during the week.

I don't think the hesitation to push to prod Friday afternoons has anything to do with the working schedule of your customers. It's more about whether you'll be able to roll back on a weekend.

Why wouldn't you be able to rollback on a weekend like any other day?

Not a family man, I see?

That has nothing to do with it. I don't like working weekends anymore that anyone else. However, every now and then it is necessary for certain systems. Also, if your rollback procedure is go, it isn't issue.

  >I don't like working weekend [sic] anymore [sic] that [sic] anyone else.

So why the feigned confusion above? It seems you already knew the answer to your own 'question.' :P

You can, but it also means you might have to.

cause I don't want to work on the weekend. Why would I introduce that risk?

Or if you need more resources, can be a lot harder on a friday evening or weekend to get ahold of someone else to help.

Assuming they don’t have an actual life and aren’t somewhere touching grass.

I’ve worked with several people who either confessed to or were accused of having taken up pastimes that would guarantee they weren’t available during times they shouldn’t be expected to be available.

Management gets rewarded for getting more work out of us and there’s always someone pushing boundaries trying to leach a little more work out of the staff they have. These same people seem to think burnout is a big mystery, because to admit they caused the problem would be to admit they did something wrong.

It’s ok, claude can do it for you.

Thursday is pretty close to the weekend. Too risky on Thursday?

I'm old, I do prod changes Monday to Wednesday for that reason. I'm not paid to work on weekends. Sure, you can argue that people should be competent enough to never release software with a bug, but I sometimes fuck up.

Or just have a good rollback plan like a snapshot.

If the system I work on needs to be restored from a snapshot then I'll definitely be spending the weekend working. There are systems you can't just rollback and restore a snapshot and all is fine.

So when you roll a patch out on a Tuesday morning and it breaks, how much revenue do your client's lose? I'm all for telling a client to eat it because I'm not working on the weekend for some BS, but there are critical systems that CAN'T be offline during the week.

And then there’s the Tower of Hanoi of breaking major changes into tiny pieces that can increment without making rollbacks impossible. Always two more steps than you would think you need and requires patience that management often doesn’t possess.

Example, got a feature with a new required field? Add empty field, synthesize a default value for existing rows, start accepting data, validate that data, require the data, use the data.

If you’re doing deployments once a week, you end up juggling two stories so you’re not sitting on your hands the whole time, and two stories divided your attention, so the odds of fucking it up go up anyway even though you’re being careful on paper.

Meanwhile one of your coworkers is doing the same dance on another feature, so you start tripping over each other as team size increases.

Thursday is a good choice, especially if Fridays are less busy than other days as that gives the team a day to work on the issue if necessary and minimises the business impact if it goes bang.

Scrum originally recommended people run sprints from Wednesday to Wednesday so you were doing deployments in the middle of the week. I’ve never worked anywhere that did a Scrum right, and I don’t mean that in the “well then you’re doing it wrong” sense. I used Kanban before Scrum and man is that a downgrade.

Because you are patching a critical system that can't be down during the week.

In which case, why wouldn't you deploy it during the weekend? That way, you're already prepared to be working several hours on a Saturday (presumably earning overtime rates) and so it's no major issue to rollback if it goes wrong.

Friday deployments make no sense if you have teams that work Monday-Friday.

Several hours. That number goes up quickly when the bus number for something that broke unexpectedly is unavailable. While your coworker who actually knows what that error means is at the zoo with his girlfriend or with his kids.

The whole point of WLB is that we don’t live like monks.

or want to have to.

We avoided Friday deploys because we didn’t have enough usage over the weekend; traffic was oriented on M-F working hours.

It meant that issues would sit undiscovered over the weekend until Monday morning, and lead to a week starting in chaos.

Much better to get fast feedback while the deploy could be monitored against prod traffic.

We did Monday morning IST, on a build we declared valid on Friday afternoon - no pressure to declare premature victory there. The cherry on top was sprint demos first thing in the morning PST so when I first joined the project the demos got interrupted on a regular basis.

I don’t think most people fully appreciated how much work a handful of us did to make sign off faster and deployment less error prone.

So... it's 24/5, then.

Yes pendant

We have systems who’s main use in at the weekend. Other systems are only used overnight. Some the quietest part is 0930-1230 on weekdays.

The reason you don’t want to push on a Friday, or overnight, or weekend, is your experts and the top bosses tend to work m-f 9-5, so pushing early in the day and early in the week ensures more people are available to fix problems and run comms.

There is someone on call at all times at every level in my company and we are not very big. It's rare I see anything I can't fix, but if I do, it's easy to click that button that makes the alarms go off and they'll be on the call in minutes.

Is the guy on call your best troubleshooter? No. That guy doesn’t work weird hours because you’d have to be paying him $50k more to put up with that BS.

It’s easy to think things aren’t going to go to hell, but look at the huge companies that screw things up. Amazon. GitHub. It could happen to you, too.

Team America: World Police

In public utility we only did deployments on Fridays after 5PM

There's a bifurcation in best practices.

For industries that serve customer bases that are down-cycled on weekends, the dominant effect is "Deploy on Friday to minimize customer disruption chance."

For industries that operate with a 24/7 uptime expectation, the dominant effect is "do not deploy on Friday so you don't have to pay for your expensive software engineers pulling extra hours on a weekend. Much better to have disruption while the folks most qualified to fix it are in-house and alert."

When the update requires downtime (or has such a high likelihood of downtime) that you'd be knowingly (or almost knowingly) inconveniencing users, do it off-hours (e.g. nights and weekends) and gather availability commitments from all the people you anticipate might be useful. If something goes so terribly sideways that it takes a long while to stabilize given the skeleton crew, at least you have plenty of time before users need it again. Friday evening maximizes that window.

Else, do it during regular business hours and very far from time off (whether that's CoB, weekend, PTO, etc. -- "not on Friday"). If something goes so terribly sideways that users are impacted, at least you have nearly everyone available so hopefully it won't take long to stabilize.

> Friday evening maximizes that window.

For the FAA it sure as fuck doesn’t.

Unless you have god-tier CI/CD, which most places do not, I do not like Friday changes. I’ve worked at precisely one place where it didn’t bother me, and one place where they didn’t have CD, but shared my belief and didn’t deploy on Friday.

IT in corporate is actually more panicked than IT in life safety, because life safety invests more redundant design.

the update caused the reboot but could that mean the reboot happened due to a prior update? or literally the reboot was FOR the update hmm

Windows Update decided to reboot automatically because somebody didn't press the skip button enough time?

> we couldn't even release new updates during a weekday

so you were forced to release updates on weekends? isn't that worse?

Worse for the employees. Better for the clients.

It depends on the schedules of the people who'll have to fix it if something goes wrong. I don't mind working weekends, so when I was in the sysadmin game I always tried to make significant changes late friday/early saturday, so if something went wrong I had the weekend to fix it & clean up. But if the people needed to fix a problem take weekends off, pretty much the opposite schedule applies.

Flights generally peak Mon-Thu due to business flights, weekends are usually lighter unless it's a long holiday weekend.

"3 pm on a Friday is indeed bizarre"

Most likely, they HAD to do it.

Probably was scheduled for Thursday evening but was delayed due to the Outlook issues.

Reply on news.ycombinator.com