[1] https://rethinkdb.com/assets/images/docs/performance-report/...
First, the author is looking for a characteristic that can be monitored automatically - for example, alarm if P99 latency is over 2 s. Visualizations while useful don’t help with that.
Second, the author is looking for a solution that can run in soft real time, so that it can be used for system monitoring.
Third, they’re looking for a solution that does not have to aggregate the full raw data set from across the fleet. It is implied that they are working with reasonably large fleets such that full aggregation is impractical; or maybe just too costly or too slow.
If you were able to aggregate the full raw data set in real time and compute the Nth percentile, then that statistic would meet the author’s needs. Their point is that actually computing the Nth percentile is expensive and not commonly done in real-time monitoring (hence the statistic is usually an average of host-level Nth percentile).
The challenge they’ve proposed is to define a statistic that is more useful for alarming while still avoiding the need to aggregate the entire raw data set.
I thought this was a thoughtful article with a clever suggestion. “Percent of requests over threshold” meets these criteria. One criticism of this approach however is that the threshold needs to be known ahead of time, prior to aggregation.
Store counts of numbers of events in each bucket.
A few hundred integers per server isn't hard to store and aggregate.
Prometheus does this out of the box.
Now you can recreate any of the charts you want!
Histograms can absolutely be used for alerting. We have done this at Circonus for ages: https://www.circonus.com/features/analytics/
I wrote up the case of latency monitoring 6 weeks ago here: https://www.circonus.com/2018/08/30/latency-slos-done-right/
As long as your histogram has the final long tail bucket (>99%) included, you'll be fine.
After all, the 100'th percentile latencies are what your users will experience as the worst case. That's what they will perceive and remember. That matters for usability. While there is no sane way to ever eliminate the most obscene outliers, you can target the worst-case behaviour and find ways to limit how badly it impacts your users.
Anecdote from work: our exchange team (who routinely consider 4ms for service response too slow) monitor p99 for general performance and p100 for the nastiest outliers. They want to know exactly how bad the performance is for the observed worst-case scenarios.