Else, do it during regular business hours and very far from time off (whether that's CoB, weekend, PTO, etc. -- "not on Friday"). If something goes so terribly sideways that users are impacted, at least you have nearly everyone available so hopefully it won't take long to stabilize.
For the FAA it sure as fuck doesn’t.
It meant that issues would sit undiscovered over the weekend until Monday morning, and lead to a week starting in chaos.
Much better to get fast feedback while the deploy could be monitored against prod traffic.
I don’t think most people fully appreciated how much work a handful of us did to make sign off faster and deployment less error prone.
I’ve worked with several people who either confessed to or were accused of having taken up pastimes that would guarantee they weren’t available during times they shouldn’t be expected to be available.
Management gets rewarded for getting more work out of us and there’s always someone pushing boundaries trying to leach a little more work out of the staff they have. These same people seem to think burnout is a big mystery, because to admit they caused the problem would be to admit they did something wrong.
Friday deployments make no sense if you have teams that work Monday-Friday.
The whole point of WLB is that we don’t live like monks.
Example, got a feature with a new required field? Add empty field, synthesize a default value for existing rows, start accepting data, validate that data, require the data, use the data.
If you’re doing deployments once a week, you end up juggling two stories so you’re not sitting on your hands the whole time, and two stories divided your attention, so the odds of fucking it up go up anyway even though you’re being careful on paper.
Meanwhile one of your coworkers is doing the same dance on another feature, so you start tripping over each other as team size increases.
>I don't like working weekend [sic] anymore [sic] that [sic] anyone else.
The reason you don’t want to push on a Friday, or overnight, or weekend, is your experts and the top bosses tend to work m-f 9-5, so pushing early in the day and early in the week ensures more people are available to fix problems and run comms.
It’s easy to think things aren’t going to go to hell, but look at the huge companies that screw things up. Amazon. GitHub. It could happen to you, too.
For industries that serve customer bases that are down-cycled on weekends, the dominant effect is "Deploy on Friday to minimize customer disruption chance."
For industries that operate with a 24/7 uptime expectation, the dominant effect is "do not deploy on Friday so you don't have to pay for your expensive software engineers pulling extra hours on a weekend. Much better to have disruption while the folks most qualified to fix it are in-house and alert."
If something goes down on Saturday there’s a good chance I’m out of cell service
Hey software engineers - you know that working weekends is unusual, right? Like... please tell me someone has at least negotiated your right to only work 5 days a week so if you're working a weekend you get Monday and Tuesday off.
If not... Has anyone told y'all you should unionize?
What seems to be unique in American companies is you’re expected to work shifts or nights but also work m-f 9-5 at the same time, which is really weird.
Same with of call, if you get called at 3am on an on/call rota for a specialised third line escalation many companies still expect you to work the next day, and they might not even pay you for being on call?!
The other option is to fail during the workweek. If the software is directly generating money for the users and the developers are judged on what actual business value they provide then failure during business hours is worse.
People literally fought, bled, and died for our right to not do capitalism 24/7/365, because with no back-pressure that's what the efficiency engine will demand of us.
People fought hard for the 40 hour work week and 2 day weekend.
You think that should be given up?
so you were forced to release updates on weekends? isn't that worse?
Most likely, they HAD to do it.
But what I’m hearing is that one company “L3Harris” could take down the airfristructure anytime it wanted to. That’s not great.
Well, it worked so far, I think it is great. 100% availability is a dream.
That’s why for example, you have ATC Zero protocols when communications go out all of a sudden at airports. And I imagine there is something similar for airborne planes that are not near any of their destinations.
It's been pretty cool to see NASA lock in under the new administrator, starr making things in house and not wait on contractors etc.
Foreign cyber-adversaries (the usual suspects ;) are taking notes.
It wouldn't be my pick, that's for sure.
My (hopeful) assumption is that air traffic control isn't based on the "weekends are the off hours" mindset.
So the deployment target seems like it should have been Monday @ 10pm. (Or whenever the bulk of flights have landed, but there's still enough for a validation test)
Overnight periods are when the air travel system largely resets itself from each day's cascading delays, mechanical failures and bad weather.
I worked for a person who had a fixation on "neat" deadines: the "first of the month", "the end of the week", "the start of the quarter", etc. It'd be one thing if the dates were tied to legal compliance (tax filings, etc), but that usually wasn't the case. They were chosen because they were "tidy".
I'd time meetings looking to maximize time for work. You try to meet late in the day Wednesday because an early Tuesday meeting might get an "end of the week" deadline. A late Wednesday meeting was more likely to garner a "end of next week" deadline. Meeting Thursday or Friday guaranteed an "end of next week" but you lost Thursday and Friday work time. Wednesday was a gamble, though, because it was still early enough to get you an "end of the week". Meeting after lunch helped with that.
You always had to be aware of the numeric calendar day, though. Gunning for an "end of next week" deadline when next Wednesday was the end of the month (or worse, end of the quarter) almost guaranteed you an earlier deadline than you wanted.
All of this deadline-setting happened with little regard for the amount of time work was going to take.
It was immensely fun, when setting a deadline in this person's absence, to get a group to agree on a mid-week deadline. You could see it make this person's skin crawl hearing that the work would be done on Tuesday or a Thursday.
I spend at least 1/3 of my fridays answering all requests with "Read Only Friday."
If they vibe coded this code, that very likely crosses into criminal charges. That would mean they did not properly vet the code or sign off with human reviewers.
This is the type of corner cutting that caused boeing to kill people when they bypassed development and testing procedures that were required to improve safety.
By people with unreliable systems and a lot of myhts behind.
In a well engineered organization, you can deploy whenever you want. Sure, it will not work 100% of the time, but you figure out why, fix the problem and you will be one step closer to the mythical 100%. And so on. Deploying less frequently is what creates more problems.
But the entire first months of the Iran War was started on Friday after the markets closed. It took a few weeks for most Americans to catch on.
My money is on some mundane ops issue like "it takes lots of time to get sign off for the upgrade from lots of different stakeholders and the necessary upgrade was falling days or weeks behind, leading to a key IT person to decide to YOLO the upgrade/deployment".
I've heard that Sec Transport Duffy has been pushing FAA upgrades faster than is safe, so maybe the employee rushed it so this upgrade wouldn't hold back some other larger timeline. There have been several articles recently about how the FAA is full of people who are "too risk averse", suggesting that there might be an effort to push riskier projects faster. Having a nationwide outage is one of the likely symptoms of raising risk tolerance.
In what sense? It was well reported at the time, so presumably most people know about it. I'd be interested in what makes you think the time and day of week chosen would delay market or public reaction be _weeks_.
As an insider to the technology industry, I would surmise the technical reason for it is that they didn't want to work over the weekend.
(Yes, this comment is tongue in cheek, but not everyone follows the "don't deploy on Friday" rule, even people in mission critical fields.)
Are they unionized? Maybe management didn't want to pay time-and-a-half(+) for weekend work?
Isn't that exactly what they did? If you do major change on Friday at 3pm you likely will work over weekend.
It’s just dumb.
If it was a malicious act, or maybe even two, would they publicly state it was the case ?
I really hope we get an official or leaked root cause analysis and postmortem of this and it doesn't get buried because all the money in my pockets is willing to bet there was some deep incompetence responsible at the executive level.