-1: if the fix breaks customers in other ways, they're screwed as well. Damned if you do, damned if you don't...
Meanwhile, WAFs are being continuously updated with detection signatures.
As for WAF rules, those can usually be flipped on and off with a single switch.
The trouble is if the WAF change e.g. messes up a redirect or turns a success response into an error response or vice versa, which confuses some other system into giving multiple users the same session cookie and gives them access to one another's accounts or writes corrupt entries to the database or whatever it is. You can revert the rule but that doesn't revert the state changes that have already happened.
There is inherently a trade off between keeping the system online, keeping the system patched and testing the patch before applying it.
Sometimes. In a world in which customers are increasingly adopting immutable infrastructure patterns and (contrary to best practice) using Launch Templates with symbolic references to `latest` versions of AMIs, this isn't always easy.
> There is inherently a trade off between keeping the system online, keeping the system patched and testing the patch before applying it.
On this we agree 100%.
Any vendor that delivers updated packages without thorough regression testing that ends up causing an incident for their customers as a result will get a thrashing from the community. Remember the CrowdStrike BSOD incident in 2024?
Maybe Ubuntu and others are cutting corners to deliver more quickly. It's their risk to take, but it's still a risk.
If you need to test it you should pin packages.
Unfortunately people rarely follow best practices.
Customers frequently do risky things like using launch templates that refer to the latest AMI. AMI changes, Auto Scaling launches a new/replacement instance in the fleet without adequate testing, *boom*.