Then we started moving non-database things to Y. Each thing we would move would first still point to the databases on X, then would be updated so that it is reading from the mirror at Y but still writing to X. Once everything was working that way, we changed the configuration so we could write to the databases on either X or Y, with two way replication between the two, and then started changing things at Y to write to the Y databases. When that was done we had everything running on Y, but with the two way replication between X and Y still going so that if we found a serious problem we could revert the problem application back to X.
We than ran this way for a while until we were happy with the results. The powers that be then authorized shutting down everything at X. Since we had sensitive data at X (such as onfile credit cards), we needed to make sure we deleted everything before relinquishing the servers. Deleting the data was my task. I got the go ahead to start, and fired up my scripts.
A little while later a high level manager comes running to my office yelling "stop the wipe!".
It turned out that the IT folks had not turned off the two way replication before turning the servers over to me for wiping. My scripts included deleting all the data from within the database, so that got replicated over to the live servers and wiped out everything in our customer database. Oops!
We were able to reconstruct it from the latest daily backup plus the replication logs, but it did lead to a few hours outage in the middle of the day.
(This wasn't actually my stuff up, as it was IT that omitted the "stop replication" step, but I had seen their migration checklist and should have caught that they did not have such a step).
2. In my college days, working part time as a sysadmin at Caltech High Energy Physics, I wrote a batch processing system for the physicists. (I don't recall why they wanted some new system instead of just using cron or at, but they did). There were two programs in my system, "batman" (BATch MANager) for the physicists to use to manage their jobs, and a dameon "robin" (Run Overnight Batch INput) to actually do the work. There was a configuration file, /etc/batman.
While working on batman and robin, I had many occasions to replace the configuration file, or to delete it. In other words, I had many occasions to type, as root, "cp new-config /etc/batman" or "rm /etc/batman".
Out of habit I had a tendency to automatically type "passwd" whenever I typed "/etc/". So one day I ended up typing "rm /etc/passwd" instead of "rm /etc/batman". Oops. I contacted the other system administrator, the one who handled backups, and he came in and restored /etc/passwd.
Then I did it again! He restored it again, and this time also made a hard link /etc/safe-from-tzs to /etc/passwd, so that if I did it again we could just relink instead of restoring from backups. This was quite embarrassing.
That worked fine, until instead of it being an rm command that I did the batman/passwd switch on it it was a cp command. That led to another safe-from-tzs that was a copy, not a link. Even more embarrassing.
(I know that there are a couple more incidents I left out, but I don't remember the timing. One of them I was able to recover without bothering the other sysadmin, because it was an early Sunday morning and I was sure no one else was on and no batch processes were running, and I was able to cut power to the VAX before the periodic cache flush committed my error to disk).
3. This one wasn't mine, but I had to figure out what happened and fix it. The place I worked sold a subscription product. One day the re-billing somehow ended up getting big runs of off by ones on the credit cards. What I mean by that is instead of charging account A1's credit card, C1, and charging account A2's credit card, C2, and so on, we ended up putting A1's charge on C2, A2's charge on C3, and so on.
Fortunately we only had a small number of subscription plans, and one was by far the most popular. That resulted in most of these shifted charges still putting the right amount on most cards, so only a few transactions had to be voided and redone due to the wrong amount. We got very lucky there.
I was given the task of figuring out what had gone wrong, since the billing process and databases all ran on Unix systems and I had by far the most Unix experience (I'd been a Unix kernel hacker at Interactive System Corporation and at Callan Data Systems in prior lives). I fairly quickly found the problem. The billing process was forking off children so that it could process several accounts in parallel. That was fine, except that it was opening its database connections in the parent process before the fork. Different children ended up sharing a single database connection, and so one child could end up reading results there were from another child's query. I restructured it a bit so that it didn't need to use the database until after the fork, and moved the database opening into the children and all was well.