Yeah. That guy should not be allowed anywhere near the production workloads. "I solve problems", my ass.
Otherwise, I'd never dare to do something like that.
And I'm not so crazy as to do such an operation without the appropriate tests and foundations. Of course, when I started, I had all the conditions to be able to do it, and I had already conducted all possible tests. :-)
This is not possible with a client's workload unless you can actually test it. That's why AWS will warn you multiple times if they need to migrate your EC2 instance onto a different hardware node. Even if it is technically "better".
Of course, the fact that clients trust their workloads to this guy probably means that there was nothing important there.
The VM change was sufficient enough to alter the runtime of a task by several times. This is NOT a small inconsequential change.
You _have_ to warn your clients when you do stuff like this.
In every system I’ve worked on, the agreement is in terms of an SLO. We never gave our customers any sort of expectation (or guarantee) that we wouldn’t suddenly wildly beat our SLO targets (and, in fact, we often did, due to routine upgrades).
Having said that, certain customers dictate production freezes during launches, or only want to run stuff that’s been baked in production elsewhere for 3-6 months. Upgrading those customers behind their backs would be unacceptable, especially because they pay extra for a crappier but more stable setup.
And he found it out only by talking to the customer, indicating approximately zero testing from his side.
It might have gone the other way easily.