Any insightful takeaways from "watching the logs" and "correcting issues" in terms of what you would do differently?
More generally, when you claim in system design that you have invariants, if you aren't using automation to enforce your invariants, they will inevitably be bypassed to your detriment by eager and/or lazy engineers. You will find this out at the worst possible time.