It pretty much boils down to: it depends upon what the business wants to prioritize; operating margin or resiliency. There is an entire subfield investigating the statistical foundations of resiliency, and the general case of N-modular redundancy is in practice implemented as triple modular redundancy in most commercial systems that want to spend in this vector.
> Like it or not, these highly trained individuals are going to make mistakes every now and then.
Absolutely, and here is where the organization's no-blame learning culture swings into action for the well-led teams.
> It's a lot harder than it looks.
We all know this, and we can all help each other get better to deliver ever increasing value to our customers by sharing what works for the context we deployed within!