I had already resigned. First day of the last two weeks. I was updating a firewall rules in Cloudpassage. Their UI sucks, and doesn't make it clear what changes were made when you click "save". Apparently, I had accidentally changed port 443 from allow to block.

Well, when the site goes down the CEO charges into the DevOps eng room and starts screaming "we're under DDOS attack!" This was his go-to cause of any problem ever, and of course is never true.

Well anyway, with all the screaming and ruckus I had forgotten that I was just in Cloudpassage a few minutes prior changing something unrelated. So we are investigating the problem for almost an hour.

As services in AWS autoscale - they work for about 15 seconds, and then are no longer reachable. That's when I realize....once cloudpassage updates the firewall rules, the instance becomes blocked. Doh. I switched it back and everything went back to normal.

Two years without incident and then boom, ruined in the last two weeks. Felt awful, but fortunately they were cool about it.