Post Mortem of the incident discussed here: https://news.ycombinator.com/item?id=26581027