Hacker News Reader

Comment by KaiserPro | Hacker News Reader

Grafana and graphite are what makes managing large infrastructure manageable.

I was introduced to grafana in early 2014. I was a bit sceptical as I was using graphitus to make dashboards. However I soon converted.

I maintained a very large graphite cluster at the Financial Times (I think it was about 1 million active metrics, but it might have be 0.5 mill, I forget) The only sane way to manage the front end was using grafana. Simple oauth2 integration meant that I could avoid the nightmare of trying to get AD access, and it also mean't one click SSO.

Grafana was one of those tools that was self evidently the best in class, so it was widely adopted. Within two years, virtually every team screen had grafana on it. Non programmers used it, and even set alerts. How many other "devop" tool can boast that level of universality?

Either way, keep up the good work, and best of luck.

View on HN

As someone that recently consulted for a F500 client and had to make recommendations regarding their Grafana instance (among other things), we noted they were curating too many metrics (somewhere in the thousands). Our belief being that if you're providing stakeholders with so many metrics, you're forcing them to make their own decisions regarding what's valuable to track and what's not - rather than allowing leadership to provide direction as to how they're measuring performance, etc.

I can't imagine what it'd be like (as a stakeholder), using a Grafana instance that, in total, has >500k metrics. Would assume many of those are depreciated/ do not provide any value/ or do not spur any action by stakeholders.

the graphite instance had the metrics, not grafana. I don't know how many were actually graphed. One thing that I can be sure of was that they'd all been pushed within the last week, otherwise they'd get deleted.

There were at the time about 200 dashboards. They were controlled and curated by their own teams. It was pretty much the only shared tool that worked well. The only thing that I encouraged was tagging, but even then, they mostly did it themselves to make finding things easier.

There were about 80 active products, most had _a_ dashboard.

The cruicial thing was that it doesn't cost much to record those metrics. This means that post incident we can easily put an alert in, or prove x affects y because z.

limiting the number of metrics recorded is frankly silly. Enforcing rules about quality and location, certainly, its something I spend a reasonable amount of time on.

for example, the front end was a microservice. Each http call of each microservice was graphed, which allowed quick and simple diagnostics for general performance. Most of the time its not needed, but when you _do_ need it, its critical to have context

I've worked with similar scale, and the situation is basically that when you have a thousand people working on a service, they have different needs. Ops needs a variety of host and container level metrics. Not just for action by stakeholders, but for autoscaling, autoremediation, etc. If you have a few thousand servers, you're probably talking 100k metrics right off the bat. More if you want statsd aggregate metrics instead of just one summary stat.

And if you have microservices, you want to track how well each client-server pair is doing, on both sides of the equation, which means tracking error codes, success/fail rates, etc.

Finance wants its own metrics to measure capacity versus utilization to prove to the CFO the spending is appropriately constrained.

Devs want to prove their system works and works quickly, so you'll have a variety of metrics revolving around subcomponent usage, and performance timing. Maybe even cache rates.

Not all of these metrics will spark action by stakeholders. Some will be retained 'just in case' since you can't retroactively collect data. When perf drops, in a canary because GC pauses are increasing, you definitely want to be able see both performance metrics over time as well as GC metrics.

You need to write up a how to, and do an ama...

Reply on news.ycombinator.com