Going back to your comment, I feel like I missed this part, or at least did not address it:

>Write better tests, clean-up the interfaces, you can even try to use formal methods to ensure correctness etc.

I definitely put effort into all these things, but my experience shows that no matter how much work goes into that, things will find a way to fail in a way I could not predict.