netingle

Definitely! I'd recommend using variables so you don't have to replicate the dashboards, but they don't work with alerts (yet). You could also use something like grafonnet or grafana-lib to generate dashboards with code.

Lots of options - Watch this space!

Thank you! I can't claim any credit for the Grafana project though - Torkel & team have done and continue to do an excellent job there. I mostly work on Prometheus, Cortex & Loki.

>50% of out engineering effort goes into Grafana - but we've already branched out into other spaces, with a team working on Metrictank (scalable Graphite), Cortex (scalable Prometheus), Loki (log aggregation) and even the Graphite and Prometheus projects.

That being said, Grafana will always treat other datasources as first class citizens, like we already do with Influx, MySQL, CloudWatch, Stackdriver, Elastic etc. This is our "un-vendor" approach.

We mean distributed traces ala zipkin, jaeger etc.

Thats a very good question! The opensource Loki project will always be fully featured, but we are looking at building an enterprise version in a very similar way to Grafana Enterprise.

Having the right differentiation is key here, and something we're super sensitive to - for instance, we put LDAP (a typically enterprise feature) into OSS Grafana. What this space and let us know what you think!

100% yes - this is a hill I will die on.

I got it wrong with Cortex (scalable Prometheus) and have been paying the price - poor adoption, smaller community and less mindshare. We're 100% committed to making sure Loki and Grafana are super easy to self-host, and are even putting time and effort into making Cortex and Metrictank easier too.

Yeah! Have you tried https://grafana.com/grafana/plugins/natel-plotly-panel? I'm pretty sure Ryan, mentioned in the article, wrote that....

So would I! It is something we're going start work on pretty soon I think - I mainly work on Prometheus, Loki, Cortex etc and not so much on Grafana. Will let one of the Grafana team comment.

Hi! Tom from Grafana Labs here to answer any questions you might have...

Author (of Cortex and the blog post) here. AMA!

I want to know more... How does one know one is in orbit? How do you work out when to fire you thrusters to leave orbit and head towards something like the moon?

Yes we do! Plan on cutting v0.1 in the next week or so, kubecon kinda got in the way... I’ll work on adding some binaries to that - what platform you looking for?

Loki author here: got some ideas? We’re all ears!

We’ve been using jsonnet to combine Prometheus alerts and Grafana dashboards into little packages called monitoring Mixins: https://github.com/kubernetes-monitoring/kubernetes-mixin

A bunch more exist for things like Prometheus, consul, etc, vault, cluster, ceph, memcached etc

We’ve also built a super simple package manager for this called jsonnet-bundler (jb for short).

And of course we use jsonnet (with the ksonnet lib) at Grafana Labs for all our Kubernetes jobs.

> I'd expect you'd have to run an API that your customers can code against to push data to you.

Thats a good point; Cortex does accept pushes of metrics but its not that simple, and its definitely not easy right now. We're going to be working with the OpenMetrics project to have a better story about push and serverless - I expect we'll have this pretty soon, ie a matter of months.

Nice! I agree Cortex's operational overhead can be quite high. Good news! We are going to fix that. We've got PRs open to add local storage (based on BoltDB) to remove the dependancy on DynamoDB/Bigtable, and I've almost finished a single binary, non-microservices version of Cortex.

If you want a sneak peak, checkout grafana/loki; its based on Cortex and has all theses features - buts it for logs, not metrics.

I recently opened a PR to add support for pushing metrics to Cortex from Telegraf, so you don't need Prometheus as the agent: https://github.com/influxdata/telegraf/pull/5003

NB I'm the main author of Cortex...

Whatsmore, Cortex is push based...

Its coming :-)

> indexes values in each logline.

We batch together sets of lines into compressed "chunks", and only add index entries per chunk. In practices, chunks have 1000s of lines in them - so the index tends to be many orders of magnitude smaller than in other systems.

We've put a new logo up now - what do you think?

And Loki has all three ;-)

There are only two hard problems in computer science: cache invalidation and naming things.

Of course! Everything is opensource, all apache licensed:

https://github.com/grafana/loki/tree/master/pkg/promtail

There is a REST API, yes [0]. Currently you send snappy-compressed protobufs[1] containing yours log to a HTTP endpoint.

We initially started using fluentd as the agent, but we found its metadata "enrichment" facilities weren't reliable enough - we'd get log lines without the pod tags, for instance. For something like Loki, which depends really heavily on the metadata for index, this was super important. So we wrote promtail.

[0] https://github.com/grafana/loki/blob/master/docs/api.md [1] https://github.com/grafana/loki/blob/master/pkg/logproto/log...

Exactly! In fact, we're moving to use some of mtails libraries and functionality [0]. The main difference is we do centralise, scalable persistence of the logs themselves, wheres IIUC mtail just processes them and exports metrics.

Thanks for the feedback, and feel free to reach out with any questions.

[0] https://github.com/grafana/loki/pull/51

Thats a really good question. I feel like its pretty well covered in the design doc[0], and you should also read about OKlog[1] which really championed this idea.

To be clear: Loki indexes metadata about the streams, and the streams themselves are indexed by time. We don't full-text index the streams though. We think, when combined with metrics, this represents a nice trade off in terms of complexity vs features. This simplification not only makes Loki easier to understand, but also scale and operate.

[0] https://docs.google.com/document/d/11tjK_lvp1-SVsFZjgOTr1vV3... [1] https://github.com/oklog/oklog

> Is this logging already available on Grafana Cloud ? Because someone I know is looking for a hosted logging solution and was looking at ELK till now.

There is a free preview right now: http://grafana.com/loki We'll be introducing a paid tier in the next few months, once things are more stable.

> Do you have integration with Docker ? Especially Logspout ?

Not yet! That sounds like a great idea though - mind filling an issue in the repo for it so I don't forget?

Thats fair; the logo was a last minute thing, we'll work on a new one...

If all you use ELK for us simple process/container logs then yes! That’s the whole idea.

But Loki is not a replacement for Elastic - we don’t have any complex query support, and we don’t do full text search. Elastic is great for analytics or BI, but we think Loki is the way forward for your container/pod logs.