* Not dealing with null data properly
* Not testing it properly
* Not having test coverage showing your new thing is tested
* Not exercising it on a subset of prod after deployment to show it works without falling over before it gets pushed absolutely everywhere
Standards in this industry have dropped over the years, but by this much? If you had done this 10 years ago as a Google customer for something far less critical everyone on their side would be smugly lolling at you, and rightly so.
> Not dealing with null data properly
This is _hardly_ a "junior level mistake". That kind of bug is pervasive in all the languages they're likely using for this service (Go, Java, C++) written even by the most "senior" developers.
This was reading fields from a database which were coming back null, not some pointer that mutates to being null after a series of nasty state transitions, and so this is very much in the junior category.
I haven't done such a mistake in many, many years, other kinds of null issues from internal logic I'm still very much guilty of at times, but definitely not "I forgot to handle a null from a data source I read from". It's so pervasive and common that you definitely learn from your mistakes.
1) A global feature release that went everywhere at the same time
2) Null pointer dereference
3) Lack of appropriate retry policies that resulted in a thundering herd problem
All of these are absolutely standard mistakes that everyone who's worked in the industry for some time has seen numerous times. There is nothing novel here, no weird distibited system logic, no google scale, just rookie mistakes all the way
This is 100% a process problem.
Most existing software stacks (including Google's C++, Go, Java) are by no means in a position to solve this problem, but that doesn't change that it is, in fact, a problem that fundamentally can be solved using types.
Of course, that'd require a full rewrite of the service.
It doesn't help with the "yes, in theory it can be null, but in practice it never will be" case. Once you write .expect() you are crashing the service just as badly as a dereference when your assumptions turn out to be wrong.
People say this kinda thing every day. It's just as easy to say in Rust terminology as any other ecosystem.
Rust has a lot of crucial benefits but "you can't crash the process by making lazy assumptions" isn't one of them.
And when the failures are request-scoped, you're back to the outage not being global but affecting only the customers using this feature with a bad config.
We will modularize Service Control’s architecture, so the functionality is isolated and fails open. Thus, if a corresponding check fails, Service Control can still serve API requests.
It's clear that they have issues in the service itself not related to language.
Even better if the type checker specifically highlights the fact that a value can be zero, and prevents compilation of code that doesn't specifically take this possibility into account.
They rolled out a change without feature flagging, didn’t implement exponential backoffs in the clients, didn’t implement load shedding in the servers.
This is all in the google SRE book from many years ago.
For a company the size and quality of Google to be bringing down the majority of their stack with this type of error really suggests they do not implement appropriate mitigations after serious issues.
The thing is, nobody is perfect. Except armchair HN commenters on threads about FAANG outages, of course.
Amateur hour in Mountain View.