Without those "irrelevant" low level metrics for a large growing distributed system, you're not only flying blind but crashing increasingly more and more often.
Without those "irrelevant" low level metrics for a large growing distributed system, you're not only flying blind but crashing increasingly more and more often.
Crashing more often is caught by an alert that looks at total capacity is s service. Doesn't matter if it's random bit flips or OOMing nodes at first. Longer term these metrics can be useful to increase efficiency again (should i first fix random bit flips or OOMing tasks). I would not base SLA relevant alerts on too low level alerts.
Systems that only monitor the highest levels appear to function fantastically well right up to the moment they crash spectacularly. Then the forensics will show you that at lower levels there were plenty of warning signs telling you that the system was headed for the cliffs and a large number of those signs will be apparent while there was still time to do something about them. Reliable systems engineering is not something you can do just at the highest levels because of the build in resilience in intermediary levels. This is counter-intuitive but born out in countless examples of systems that look robust but aren't versus those systems that really are robust.
Priority is always the customer. The focus can change given the circumstances and the context of each issue. Sometimes the focus must be high-level, sometime it requires a low-level focus, but the priority is always the customer.