This is one of the best incident reports I think I've ever read, detailed, honest and without recrimination.

I'm reminded of something I read in a book a while back about aviation disasters "It's not the first problem that kills you, it's the second".

In this post the described system for change management is at least as good as any I've seen in production and yet a series of small problems got out of hand quickly.