I would disagree with you only that I'm not familiar with a time when premature monitoring was added that caused me a lot of pain. If anything, the opposite has been true. Can you explain what you mean there?
I would disagree with you only that I'm not familiar with a time when premature monitoring was added that caused me a lot of pain. If anything, the opposite has been true. Can you explain what you mean there?
There's a lot of examples where you can catch something with monitoring, but it doesn't necessarily mean that you should.
A recent one from my memory: in a SaaS product a team shipped a bug that went unnoticed for a few days. It was feature flagged, so it only affected a small fraction of customers and didn't trigger any global alerts. Now, since it didn't trigger alerts, the natural post-mortem action plan was "better monitoring". That would mean monitoring and alerting on "rate of errors by customer" (or "rate of errors by endpoint by customer", I don't remember).
Given the usage pattern of the product, it was impossible to create a global monitor like that, we'd have manually configure it for each customer (and we had thousands of those). And even then, we'd inevitably be dealing with false positives every week.
The right action plan was to learn from failure, but do nothing. We got extremely unlucky during infrastructure update, shit happens. We don't need to build a complex monitoring system that catches one bug every 5 years.