The hacker in me feels great sympathy. That is a painful bug. The software engineer in me cringes at the thought of upgrading a production system without testing in the lab first.
The system administrator in me cringes at the thought of having a production server running a hodge podge of random from-source builds of apparently everything in the stack. That's just asking to suffer. And all for the sake of maybe a 5% performance gain by having it "optimized" for the workload (and maybe a performance drop because you do something stupid, or you don't understand the software in question as well as the package builder for your OS).
Gosh Courier font with Courier text in the background image is a bit hard to read.
Ironically, the "Readability" site uses text embedded in images. Needless to say, this is totally unreadable by many of the people most interested in the service.
So what happened to the back out plan?
That's one of the (I think) less appreciated benefits of running on a service like EC2. You can have all the plans in the world and still find yourself up the creek without a paddle when some impossible or very hard to revert upgrade goes wrong. But with EC2 it's a simple as taking a snapshot of the image prior to upgrading. Even better, you can fire up a brand new instance based off the current production image and upgrade that and only bring it on line when you're happy it's working - then just decommission the old one when you're done.
Nice tale. Another reminder to always test on dev before pushing updates.