How does something like this make it to production? I couldn't reply to any work emails for the greater part of the evening yesterday.
Adds more evidence to the case of "never use Google for your business".
Adds more evidence to the case of "never use Google for your business".
Even some basic testing on a real android device would've caught this
So it is very weird that they released this so (apparently?) a big chunk of their users
I am not saying the bug would have been prevented if the team operates differently. Brown paper bag release happens. However, the way this incident blew up suggests that the bad release made it to too many users in too short of a time, exactly the behavior incentivized by meeting (or exceeding) an OKR while having some spare error budget.
Do you believe WebView should never update because it's never OK to spend error budget?
If you "play the game", an error budget causes your releases to become riskier and riskier as the budget grows. This is optimal because a riskier release allows you to accelerate changes, allowing you to hit other OKR.
The issue is that this will eventually cause very risky releases (where you would slow down otherwise). In a sense, an error budget does eventually become "amount of intentionally deployed errors" because you are incentivized to increase risk until an error does appear.
> The solution to this Chubby scenario is interesting: SRE makes sure that global Chubby meets, but does not significantly exceed, its service level objective. In any given quarter, if a true failure has not dropped availability below the target, a controlled outage will be synthesized by intentionally taking down the system.
I doubt there are any customer-facing systems that practice this technique. Just super interesting that it exists.
[1]: https://sre.google/sre-book/service-level-objectives/#xref_r...