Hacker News Reader

Comment by chuhnk | Hacker News Reader

I once ran rm -rf in the production mysql data directory.

Shit happens.

I did that too. Destroyed out Zabbix database. Neither that Zabbix server, nor the other one monitoring the server I destroyed, could alert us that anything had gone wrong for over an hour. I finally realized it when I couldn't login...

I was able to painstakingly rebuild the server after 9 hours without anyone noticing. To this day one of my biggest fuck ups and prouder accomplishments.

Are you saying that once you break something, you should break your monitoring as well, but do it very quickly since it may be too late? :)

Yeah, if you're going to break your monitoring solution, you'd better shoot it in the head and vaporize the body.

When I worked at an ecommerce company the boss (who also was a programmer) did that too (on our master engine- not just website orders, but ebay, amazon, everything historical, inventory, RMAs.... etc etc... OOPS indeed... Yep- shit happens. *Recovered from it- was a hastle and shut down everything, but it was one of thoes whoops moments. Happens to the best of us.

Been there. Done that. No backups.

Recovered all the data by using open file handles in /proc/.

Not a fun two hours.

Shit happens. Live and learn.

Have you written a blog post about that /proc/ trick? sounds like an interesting read

Here's an article that explains it: http://archive09.linux.com/feature/58142

I don't blog, but, what teraflop posted is basically correct.

I'm going to put that on a T-shirt

This close enough? https://teespring.com/human-chaos-monkey

You are the human manifestation of "the chaos monkey"

For anyone that feel they might fall under the same category (I do!), represent: https://teespring.com/human-chaos-monkey

Reply on news.ycombinator.com