> I have personally experienced network partions happening in two ways: all nodes in the cluster being updated at the same time through Windows update and firewall rules.
I'd rather try to build a four story brick building with horse dung for masonry mortar, than run my backend on windows boxes. How many cumulative hours of horror has Windows Update unleashed on the human race?
+1
War story: an ex-client of mine was on a different continent demoing a product (involving a physical device talking to a software backend hosted by a partner company) at a neutral site to a potential new customer.
Months and months of talks and prep, dry run demos in the weeks before, everyone was quietly confident. Scope for selling lots and lots of these devices if the demo goes well, but the marketspace is fairly ruthless.
Minutes before the team are due to present, a sudden and total comms loss between the device and the backend. Cue frantic calls which due to the time zones even involved getting the CEO of the partner company out of bed. Minutes ticking by..
Potential customer not impressed, said: you have ten minutes, after that we're leaving.
As the potential customer was walking out of the building, the partner company finally tracked down the issue.
Windows Update, on a backend database server.
There will be situations where you can't, special cases where you're left with few options. If the system is that critical have 2 of everything (clusters, databases, etc.) in a way where the mirror can stay untouched and keep your business going.
And if it needs saying, something like updates or reboots should never happen truly automatically (as in the software decided when or if to do it). A human should decide how to automate that and have full control over when and if it happens.
Probably nobody consulted a sysadmin when they designed the system. *grin*
Windows 10 behaviour is another story, but I would always stand on what it's a self-inflicted wound - people blamed MS for not updating soon enough... while disabling or postponing updates for literally years. MS acted. Poorly, yes, because by that time it ascended to General Motors levels of bureaucracy[0], but again - this is another story.
[0] https://www.joelonsoftware.com/2006/06/16/my-first-billg-rev...
Sure, Windows update takes a while, but it's still so much better than whatever the hell Apple is doing. Despite the fact that I abhor removing the possibility to choose what your device does when, I think forced automatic updates are net win for computer users worldwide.
In fact, I suspect Microsoft may need to be even more forceful, because even though consumer Windows rebooted itself at the time, wannacry and friends still managed to spread across the globe using an exploit that had been patched in an important security updates months before.
There are so many other, better reasons to say it besides this or automatic updates, but Fuck Microsoft. Right in the ear.
If you're using the OS version which is designed to by used on Grandma's desktop as a server then you should expect a world of pain.
Me? I try to avoid using Windows hosts because .Net runs great under Linux and you don't need to have 5 different Windows VMs running on your dev machine to have an accurate development environment.
Patching and rebooting all your nodes in a cluster at once is not highly available.
But hey, let’s focus on windows hate instead.
I think GP acutely points out that one of the problems the author has with RabbitMQ is actually a problem with (a misconfiguration of) their underlying OS, MS Windows.
If you've been bitten by this, I understand the hate completely.
Nope. Just ubuntu deploying docker update that had problem restarting the docker service again. Simple restart fixed all of them.
But I guess people do have some bias, since on PC, windows updates are really predatory.
Mass-reboots or network outages happen for all sorts of reasons.
Power outages happen. Network switches fail. Firewalls get misconfigured. Scripts get mis-scheduled.
Windows Update is just one of many ways I've seen all-at-once reboots of servers.
I don't see how this would be better on Linux. Do Linux admins never make mistakes in cron jobs? Is Linux magically immune to switch hardware failures?
If RabbitMQ couldn't tolerate reboots, then that's on RabbitMQ, not Windows.
PS: SQL Server AlwaysOn Availability Groups will happily restart after an all-servers simultaneous reboot. ZooKeeper won't. By "design", apparently. It converts a temporary outage into a permanent one... on purpose.
When was the last time a switch just restarted to apply updates? i do not dislike Windows Update but I want to control it and apply things that are convenient to me.
When it's updated by a network team. It's a semi-regular occurrence in many workplaces.
Windows Update is not forced upon Server editions. It can be scheduled at will by administrators.
This complaint is the equivalent of saying that Linux is terrible because an admin scheduled "sudo apt-get update" to run on every node of a cluster at the same instant.
Why would admins running updates concurrently be a failure of Windows, but not a failure of Linux?
I seriously don't get this complaint, ESPECIALLY because the article makes it clear that:
1. The fault was in RabbitMQ, because I quote: "The default partition handling strategy is ignore which means to just enter the partitioned state and keep trucking along in this “split brain” mode thereby thrusting your cluster in to total chaos." [1]
2. Windows update wasn't inherently at fault: "The fix for Windows update was the ensure that nodes in the cluster are patched at different times."
I bet the default partition configuration setting of RabbitMQ is also wrong when running on Linux, and the behaviour would be identically bad if an admin scheduled updates to run concurrently on all cluster nodes.
[1] That's clown-shoes programming, which is why I'm never using RabbitMQ for anything ever. I've only heard bad things, like "the default settings result in corruption and data loss."
You can never, ever rely on people being infinitely knowledgeable and vigilant. Any attempt to do so is simply showing ignorance and indifference by yourself.
An unmarked pit in the middle of the road is a hazard. Sure, drivers in theory ought to be vigilant of such things, but fundamentally the fault is with the construction crew for not leaving out bright orange markers.
The default should be safe. The default should not be to optimised to win benchmarks to benefit the few, but to provide guarantees to provide for the many.
But the few have power, and power corrupts.
Hell, half the time that person is me.
There's a lot of valid criticism for Windows Update but I don't think this is one of them (anymore?). Part of being a system administrator dealing with Windows Server is managing updates through WSUS or something else. If that's your responsibility and Windows Servers unexpectedly "just restart" then you are not qualified for that job.
Maybe switching to Linux servers solely because of how updates are applied is reasonable. Personal preference is absolutely reason enough. But I think reaching the same level of customization+automation for updates there requires similar amounts of knowledge and time spent.
Having a server reboot on you unexpectedly is because either (1) you turned on automatic updates yourself or (2) you're running a server on a Windows client SKU, e.g.: Windows 10.
(TBH I am not sure if the author is using the right tool for the job. But everyone is an arm-chair expert and I only know MQTT from my smart home; if I reboot that daemon then a temporary loss of state simply doesn't matter)