Hacker News Reader

Comment by bt3 | Hacker News Reader

As someone that recently consulted for a F500 client and had to make recommendations regarding their Grafana instance (among other things), we noted they were curating too many metrics (somewhere in the thousands). Our belief being that if you're providing stakeholders with so many metrics, you're forcing them to make their own decisions regarding what's valuable to track and what's not - rather than allowing leadership to provide direction as to how they're measuring performance, etc.

I can't imagine what it'd be like (as a stakeholder), using a Grafana instance that, in total, has >500k metrics. Would assume many of those are depreciated/ do not provide any value/ or do not spur any action by stakeholders.

View on HN

I've worked with similar scale, and the situation is basically that when you have a thousand people working on a service, they have different needs. Ops needs a variety of host and container level metrics. Not just for action by stakeholders, but for autoscaling, autoremediation, etc. If you have a few thousand servers, you're probably talking 100k metrics right off the bat. More if you want statsd aggregate metrics instead of just one summary stat.

And if you have microservices, you want to track how well each client-server pair is doing, on both sides of the equation, which means tracking error codes, success/fail rates, etc.

Finance wants its own metrics to measure capacity versus utilization to prove to the CFO the spending is appropriately constrained.

Devs want to prove their system works and works quickly, so you'll have a variety of metrics revolving around subcomponent usage, and performance timing. Maybe even cache rates.

Not all of these metrics will spark action by stakeholders. Some will be retained 'just in case' since you can't retroactively collect data. When perf drops, in a canary because GC pauses are increasing, you definitely want to be able see both performance metrics over time as well as GC metrics.

the graphite instance had the metrics, not grafana. I don't know how many were actually graphed. One thing that I can be sure of was that they'd all been pushed within the last week, otherwise they'd get deleted.

There were at the time about 200 dashboards. They were controlled and curated by their own teams. It was pretty much the only shared tool that worked well. The only thing that I encouraged was tagging, but even then, they mostly did it themselves to make finding things easier.

There were about 80 active products, most had _a_ dashboard.

The cruicial thing was that it doesn't cost much to record those metrics. This means that post incident we can easily put an alert in, or prove x affects y because z.

limiting the number of metrics recorded is frankly silly. Enforcing rules about quality and location, certainly, its something I spend a reasonable amount of time on.

for example, the front end was a microservice. Each http call of each microservice was graphed, which allowed quick and simple diagnostics for general performance. Most of the time its not needed, but when you _do_ need it, its critical to have context

Reply on news.ycombinator.com