This is just another reason to track and alert on error rates.
It also depends how sensitive you are to failures, and required availability.
But in general this sounds more like: “you need observability on error rate”, more than anything else, and that includes notification and alerting on set thresholds that seem aberrant.
Any supervisory process, systemd, supervisord, kubernetes, etc should absolutely be making those restarts visible to the administrator so that they can resolve it.
The restart is just in hopes to keep the service available (symptoms in check) until the problem is actually fixed (disease is cured).
And we should be thankful. We don't have to be aware of every single hiccup a service experiences. Even if (and specially if) you are the one responsible for keeping it up. Modern distributed systems are far too complex for us to care about every minutia.
If something failed over, and nobody noticed, is it a problem? The answer is _maybe_. How often does that happen? Is there a pattern? Is it getting more frequent? Is that happening more often than predicted?
At work, we blow up entire VMs if they fail their health checks. They can fail for many reasons, mostly uninteresting ones. And the customers don't even notice and SRE doesn't usually care. They only become relevant once there are anomalies. When you are managing thousands of instances, self-healing is required.
Error rate may not even be impacted in a significant way when those systems restart, it is often a tiny increase in a deluge of requests. So you also need observability on self-healing events to catch trends. When self-healing fails, it tends to do so pretty catastrophically.
> and alerting on set thresholds that seem aberrant.
I wish we would remove 'thresholds' from our alerting vocabulary. Very often we end up setting simple and completely arbitrary thresholds that don't actually mean much - unless it's based on SLOs and SLAs. Generally we don't know what those thresholds are supposed to be and just guess, unless it's capacity. But even for capacity: 0% disk space free is obviously bad; what people will do is set some threshold like ("alert when 80% disk is used"). Then you get page once that threshold is crossed. Is that an emergency? I don't know, maybe it took 5 years to get there. However, if you set an alert that says "Alert me if the disk will fill up in <X> days at the current rate", you can then tell if it is an emergency or not.
I suspect that you would agree with this given the use of the word "aberrant", which to me implies anomalies. But in many contexts, people think that "alert every single time CPU utilization crosses 90%" is what we are talking about.
Somehow I've never come across this before, and it's really unlocked something for me with how I will now approach alerts
You really care about time to fill (speed), and whether your system has gone nuts and is logging faster and faster than normal (acceleration).
Disk usage is X says do little.
Disk utilization rate is Y therefore will fill by YYYYMMDD is great.
Disk utilization rate change has gone off the chart so a process crapped itself and will fill the disk in 5 mins is life saving.
We have computers to use their CPUs, RAM, and disk. Using it isn’t weird.
Detecting actual unwanted or dangerous behavior is bad.
For bonus points auto-remediate it (eg take the node offline so the workload moves to another node) and alert when your overall capacity of your service is potentially going to be impacted or is near a limit, for some version of near. Being human I much prefer “you have hours to fix this” than “I’m broken fix me”.
RED, Golden Signals, etc. there are plenty of nice frameworks for how to think about utilization and performance of your service that you can use to manage capacity and health.
CPU at 90 is not one of them.
As soon as someone suggests it I always ask for how long? On which processes? on which boxes? What is the “perfect” CPU utilization? Then make them think about why we wouldn’t want to use the resources we paid for.
Of course not. Log it, and restart.