I feel that we have to mention Stanislav Petrov now.

https://en.wikipedia.org/wiki/Stanislav_Petrov

As far as I know, critical systems like reactor control software is mathematically proven first and verified after implementation. I don’t think you can go beyond that.