Early in my Ops career I wrote a log handling script for Windows servers. It was a re-write of an existing one that pre-dated a lot of us. My version used an xml configuration file, but this iteration of the script did not properly validate the configuration items read from the file. Also, Powershell had this weird xml parsing bug that if a parent had a commented child element, it would show up as an empty child element.

So I came in one afternoon shift and my colleague said "your script destroyed two Production servers last night". These servers were critical for monitoring Police radio GPS coordinates.

"Impossible! It should only work on directories we configured it for!"

I sat there looking at horror at the debug logs, and sure enough, it read the empty child element, changed directory into "", looking for files with "" name pattern, which for Powershell resulted in C:\Windows\system32 and all files. It proceeded to zip and remove "old" files. On a reboot, they did not come back up.

To add insult to injury, when they rebuilt those servers manually, the same script with the commented out config was deployed, destroying the servers again. They realised that the script was the issue and decided not to deploy that the second time.

So I learned a lot about config validation and defensive coding in the months following the incident.