Not deploying on Friday is a red flag indicating a seriously broken process
twitter.com
twitter.com
There is some magical belief that people can understand everything whatever the size of the system, and with enough efforts and processes you can guarantee nothing bad will happen 24h after deploying new features.
I actually see this belief as a red flag, as it usually comes either from hubris or misunderstanding of the technical complexity of their stack. Interestingly even if there are exceptional people that actually have a crazy good understanding of everything, the org is probably heavily relying on them, and when they're gone the culture left won't be kind to those who aren't exceptional.
PS: one can assume that whatever happens 24h after deploy on a friday is the on-call team's problem, but that's not an attitude I'd value in a company.
If the project is launching a space ship, then investment in every possible testing method is probably worth it. But when maintaining a blog, you can keep costs down if it can wait until Monday.
The idea that you can guarantee there's never any bugs in your code is incredibly naïve
There is no way you, or anyone, can guarantee that everything you do in the next N changes will go perfectly. There will be some aspect of drift your processes, tooling, and people all miss.
We follow this rule, generally, erring towards 'no' to Friday deployments. Exceptions are made for business-as-usual work or security items with a time limit.
Just make a reasonable calculated decision. What's to gain vs the risk(s).
No need to be preachy and say 'fix your processes'... but I'm tempted. Mind the urgency of your widgets.
Edit: looking over their profile, I get the feeling this is a bit of light gaslighting. They argue in bad faith, stretching the straw man so hard he's spread across the field.
They're incentivized by a crowd of people feeling inadequate, perhaps even impressed; consequently flocking to their 'education'/consultation
At the very least this reads like someone who has lead a few too many meetings and hasn't put their hands to the keyboard for anything serious in a long time.
Good luck to whoever works under this person. Doesn't seem that they understand what it is to have a life outside of work.
There's the author's glib and smug strawman response to similar common sense.
I actually agree with their sentiment, using CI and feature flags should mean having more confidence where one could choose to ship on Friday.
I also agree with the fact that eventually the team will ship a bug. Better to do so when you're likely to see it during the workweek.
Yes, you have canaries. Yes, you have processes for when things go wrong. However, at bottom, when something goes awry, you need a human overseeing the unwind.
If something goes wrong after a Friday deploy, you will have to unwind it on a Friday afternoon or weekend. It sucks to be the person dealing with that when, for example, your child has a recital, thanks.
I subscribe to "never say never", but ... Honestly, your feature is NEVER important enough that it couldn't wait until Monday.
(The exception to this is when things are "all hands on deck" for some horrible reason. However, at that point, all your safety systems have probably been deactivated, and you are so far outside of your normal release process that you are deploying immediately for everything.)
The thing is, that for big deployments you can be never so sure.
There may always be "human factor" that will cause users to click the wrong button without looking at its label.
There is no 100% certainty, no matter how much you test due to the state space.
Not deploying on Fridays and before holidays is acknowledging this and valuing your coworkers’ life outside of work.
Even if the chances something will go wrong are very slim and you have lots of really great tests, they are still non-zero. Lots of unexpected things can go wrong in modern interconnected systems with lots of network dependencies.
Not deploying on Fridays is not a matter of confidence. It's recognition that the downside is huge (losing your weekend) and the payoff is small for the vast majority of changes you could deploy. That's it.
That's because human beings and their lives matter.
I've worked in multiple industries and few of them are so critical that a launch or cutover can't wait a couple days.
And telecoms in general, have a higher level of reliability than a (squints) "agile-friendly implementation technology like microservices and incremental/evolutionary architecture " consultancy...
Unless it is an urgent bug fix, there is no reason to push it
Once we deployed on a Friday, but a partner's API also deployed, errors start happening at 2am, the API team only has generic support on call, so our devs keep investigating, logs and changes. In the end everything was OK on our side, it was the integration API that broke, by coincidence, and they were insisting it was us.
So you cannot always control, or know, when your partners deploy. But if you avoid Friday your team can sleep assure it is not their fault
There might be an unconscious bias to deploy on Friday because people want to feel they were productive that week, or manager's pressure to fulfill imaginary targets. So people might rush it just a little bit
There was a business that our system supported that had a peak transaction day on Mondays (55% of the weekly volume flow) and Fridays had less than 7% of weekly volume flow. The rest of the days had a tapering of volumes between these levels. No volumes flowed in during the weekends, but downstream processing occurred on Saturdays (of the transactions that came in Thursday/Friday that still remained).
We deployed on Thursdays. Never on Sundays (simply because Monday was a peak day and we would never recover from business SLA failures if things went wrong when we opened the Monday).
It gave our support engineers a weekday to look into issues, if any and escalate to a developer on Friday, if required.
There is such a thing called pragmatism.
Deploying something buggy on a Monday is a very different situation from deploying something buggy at 5pm on Friday, though. Most of the time your customers aren’t going to care about getting a feature 3 days early, but your team WILL care that their weekend just got eaten up for no reason.
I confess I agree, though. Thing is, there are a lot of red flags. And life is too whatever to think this red flag is a hill to die on.
Will it be a reason not to work somewhere? No. Of course not. Is it susp? Absolutely.
Gotta choose your battles carefully - any one battle could probably consume 100% of your time.
For isntance if your whole company is at a yearly retreat in some vacation park, including part of the SRE team, but two or more devs are still working for whatever reasons, would they ship a full refactoring of some service ?
I'm always interested in where the boundaries are, if you could share.
Embrace “good enough” and collect your payment. You aren’t changing the world.
In both cases I'm ready to patch whatever comes up. But the excpectation is different. So I keep less slack for the well tended depolys.