The older I get and the more systems I encounter, the more I'm convinced everything is in a state of failing. The question is only how long before it becomes noticable.
What they don't realise is that, thanks to the FFT, 2 of those 4 things actually failed over a year ago and the system has been continuing to work ever since without anyone ever noticing. One of the other things actually fails intermittently a couple of times per month, and while a couple of people have noticed that and reported it, either it's not been investigated at all, or when it was investigated everything looked fine and the report was put down as an unexplained glitch. All that was actually needed was for the 4th thing to fail, to cause this "can't happen" failure mode to suddenly ruin everyone's day.
Just because you're paying someone to go check and they don't find anything 99% of the time, you don't get rid of that. If you do, there goes the pnly sensor capable of propagating that signal to the rest of the control network. Applies equally well in electronics/circuits/or human domains.
The right level of logging is usually enough to be annoying but not so annoying you need to reduce it. That means you're hopefully thorough enough in the logging that you're reporting what you care about and the things you see are the the things you can manually fix or ignore, and fit a frequency level you can live with.
Until the stuff you're managing becomes too large for that to make sense anymore, because that only scales so far (but surprisingly far for diligent admins). Then you need to move to something more complex, which is a big job and requires an entirely different way of reviewing (like Prometheus). You're probably better doing that from the beginning, but it's time and effort but everyone can take, or may not have existed at that time.
Surprisingly far. We log terabytes of data a day, most of it is unnecessary. I've been trying to tell people that it is not a good idea but so far I've had limited success. It's easy for people to just "printf" everything (we are lucky if logging libraries are used with proper levels).
Right now, we have the most expensive 'add' operation in history. K8s pod logs some data. That is eventually noticed by fluentd, which is watching the filesystem. Fluentd sends to our logging servers over the network. That goes to kafka, then is shipped to the system that actually does the indexing. And then, there's a query running, which will then count the number of times that string appeared and updates a counter.
All of which could have been avoided if the apps just added +1 to a counter, and exposed that as /metrics so Prometheus could scrape.
Generally though I've found once you start aggregating logs you find many more uses for them.