Google’s Gmail Outage Is a Sign of Things to Come
businessweek.com
businessweek.com
> The search giant reported that it conducted an update of its load-balancing software from 8:45 a.m. to 9:13 a.m. U.S. West Coast time, and after the problems were detected it managed to quickly roll back the buggy code. But this didn’t stop some people from questioning why Google would roll out a software update during peak e-mail hours on the West Coast.
Perhaps because the update wasn't to Gmail, it was to the load balancing system? It's not uncommon to have issues arise in load balancing while under load. I'd say the engineers did a good job dealing with the problem all things considered--load balancing issues are notoriously tricky.
And by "best", I don't mean all stars (a la Fitzpatrick, Norvig, Pike or other Google celebs), but the engineer who actually followed the whole project/patch from dev environment to production.
or to put it simple: not at 1700 on friday.
The overwhelming vast majority of the world's population lives between approximately 120 west longitude and 120 east longitude. That's just 2/3 of Earth. If you take a globe and orient it so you are staring at the middle of the Pacific Ocean, you won't see very much land.
This means that even sites that are truly global should still have a strong 24 hour cyclic variation in their traffic.
Not the most focused article, but kind of cool that they wrote about such a thing. Though the headline feels like pure linkbait.
They also missed my personal favorite attribute of continuous deployment: isolated problems. If you deploy 10 features and something breaks, unless they're completely orthogonal you now have more than one place to look/person to ask/team to deal with. If you deploy one and it breaks, you know where the problem lies.
Why? GigaOm writes about that sort of thing all the time. (BusinessWeek is just syndicating here, it's a GigaOm article: http://gigaom.com/cloud/why-you-should-expect-more-online-ou... )