You haven't lived until you unwittingly run rm -rf on a nfs mount of / on a remote box. Which happens to be the fileserver for a trading shop. In the middle of trading hours.
You haven't lived until you unwittingly run rm -rf on a nfs mount of / on a remote box. Which happens to be the fileserver for a trading shop. In the middle of trading hours.
Pity he'd not considered that the old one had nfs mounts to the live data...
Thankfully we had two machines which were virtually identical OS-wise (RedHat 6 if memory serves). I was able to get everything put back from the twin machine and keep everybody happy.
Thankfully that server kept running with relatively little issue the entire time even with all those core OS files gone. I don't think any customers were at all aware.
The extent to which new calls to deleted files are made will have a strong impact on this.
I seem to remember having a problem where I'd rm-ed a file I needed and using lsof to find the file handle and then being able to cat the data in to a new file using the handle instead of the filename? Details pretty fuzzy, sorry.
Edit:
Example of recovery this way, tested now, works for me - http://pastebin.com/c2djEcqr - the crucial part that I was forgetting is that there needs to be a file handle somewhere that's still open, which is probably not true in most cases. Worth a quick check before unmounting.
I've used ddrescue and photorec for these sort of "issues" before with much success.
2) Run to comms room and yank out power cord.
3) Spend several days piecing files back together from backups and the remnant data on disk.
4) Learned a new respect for rm.