> the biggest problem we face is maintaining an understanding of what is happening in our system

This really resonated with me. Distributed systems are _hard_ and tests (while valuable) are no substitute for being able to reason about what your application is doing and what might be affecting it.