Back around 2000, I was writing software patches for a large telecom company. These patches had to be applied to live, inservice telephone switches and equipment that were powering public phone services - restarts and downtimes were not an option. Needless to say, careful consideration had to be given with regard to how a patch was developed and applied.

I don't recall the specifics, but one fateful patch had a very large compound boolean expression that included an invalid condition. For whatever reason, that expression didn't or couldn't receive 100% test coverage.

When applied, this patch took down the entire phone service (including emergency 911) for the island of Newfoundland Canada and all its residents (maybe 300-500k people).

The immediate fix was simply a new patch that reverted the bad patch. Unfortunately, it took several hours from the point we learned of the issue to writing, delivering and applying the fix.

I've heard that Newfoundlanders have a reputation for being very friendly, so I like to believe that all has been forgiven and forgotten and that I may one day visit the island in peace.