Hacker News Reader

Comment by LaserToy | Hacker News Reader

I would add - you should start your monitoring with business metrics. Monitoring low level things is good to have but putting whole emphasis on it is missing the whole point. You should be able to answer at any point of time whether users are having a problem, what problem, how many users, what are they doing to workaround?

In other words, when person is in ER, doctors are looking for heartbeat, temperature, ... , not for some low level metric, like how many grams of oxygen is consumed by some specific cell.

View on HN

Yes, in the ER, doctors look at things like heart rate, respiration rate, and temperature. But they also draw blood for electrolytes, glucose, creatine kinase, etc. since those can detect underlying problems which the body is compensating for.

A well designed distributed system is going to be able handle a certain failure rate in its components because requests will be retried automatically. If a component's failure rate increases from 0.01% to 0.1%, there will probably not be any user-visible impact... but if you can detect that increase, you might be able to correct the underlying issue before that component's failure rate increases to 1% or 10% or 100% -- at which point no amount of retrying will avoid problems.

I;m not saying you should not do that, I'm saying your focus should not be there. When things go south your CEO rarely is interested in CPU utilization, or error rate. The question they usually ask: How bad is it? What is user impact? Is it all hand on deck or it is just a glitch. When it is cascading, which system to fix first? Root cause? Component failure rate just doesn't have enough context.

And yes, distributed systems are hard, because it is inherently hard to reason about what will happen when something changes/fails.

I'm not from Uber but I recently worked (was responsible for a huge chunk of infra) at a company that is bigger and has more products running and I saw some hilarious failures. And what is different, it was rarely a bad code push, and when it was, due to the nature of the business, sometimes it was really hard to roll back.

You're missing the forest for the trees. The deeper metrics are important for diagnosis. The business metrics are what you care about.

I'd phrase your advice differently as well: monitor on contact points. Monitor where rubber meets road. Monitor where two components interesect, be it between teams working on frontend and backend components, your application and it's database, or low level metrics - the system API. I've found many interesting problems and averted several potential crises because I saw that the metrics things like requests looked different between the client and server, the system metrics and my applications, etc. Contact points are what's important, and it just happens that every (Figuratively) application has a contact point with the OS.

But that's my point: The deeper metrics aren't merely useful for diagnosis after the business metrics go sideways; they can be useful as leading indicators, to warn you before problems reach the point of affecting the business metrics which you care about.

The deep metrics, as you call them, get vanishingly irrelevant the larger the system gets, at least for the use case you mention.

To put it unscientifically, something always goes wrong, but that does not mean SLA is violated. As a corrollary, you will be alerted constantly about something that might go wrong because of some signal that someone thought might lead to some outage.

At face value it seems proactive, but it's really not. Better spend time actively making the system more reliable (e.g. look at your or someone else's postmortems, or do premortem exercises).

On the contrary, low level metrics become more important the larger a system gets, since the quantity of low probability events (e.g. cosmic ray bit flips) increases.

Without those "irrelevant" low level metrics for a large growing distributed system, you're not only flying blind but crashing increasingly more and more often.

You are just chasing your own tail by looking at those metrics. Your system needs to be resilient to these random events and you must account for those, that's for sure. But you would not put your focus on those, it's still the higher level alerts that are more relevant to the business.

Crashing more often is caught by an alert that looks at total capacity is s service. Doesn't matter if it's random bit flips or OOMing nodes at first. Longer term these metrics can be useful to increase efficiency again (should i first fix random bit flips or OOMing tasks). I would not base SLA relevant alerts on too low level alerts.

This is simply wrong. Quality starts in the basement, and works its way up. So if you monitor a system at the lowest level you will be able to build confidence about those layers and you will spot trends long before they will become apparent at higher levels because those higher levels will erase the fact that at lower levels things are already going wrong.

Systems that only monitor the highest levels appear to function fantastically well right up to the moment they crash spectacularly. Then the forensics will show you that at lower levels there were plenty of warning signs telling you that the system was headed for the cliffs and a large number of those signs will be apparent while there was still time to do something about them. Reliable systems engineering is not something you can do just at the highest levels because of the build in resilience in intermediary levels. This is counter-intuitive but born out in countless examples of systems that look robust but aren't versus those systems that really are robust.

After reading this thread, everyone convinced me there are good arguments for having different levels of metrics. I think we can better understand the problem if we introduce the word "priority".

Priority is always the customer. The focus can change given the circumstances and the context of each issue. Sometimes the focus must be high-level, sometime it requires a low-level focus, but the priority is always the customer.

Reply on news.ycombinator.com