jrv

I've been using the Freenode-to-Matrix-bridged Freenode IRC rooms for around a month now and am pretty happy, besides the occasional glitch due to Matrix and its bridges still being in beta.

While Prometheus's built-in local storage is not distributed, we are in the process of implementing generic interfaces to allow Prometheus to talk to decoupled distributed, horizontally scalable, durable, remote storage (or even other non-storage systems like queues etc.). There will be several implementations sitting behind this interface in the end (see already Vulcan from DO and Frankenstein from Weaveworks, both experimental right now), depending on what you'd rather run and what fits your use case best.

Most importantly, Prometheus will never decide to exclude a feature for commercial reasons. As it's a 100% open-source and independent project not controlled by any single company, we don't have those kinds of incentives. We just want to build the best monitoring system. Note that we still say "no" to a lot of features, but that's only for technical reasons, to avoid bloat, or to keep the project maintainable.

> I think the push model is better in terms of security and discovery (which is how I think most of the other metric aggregators work).

Interesting, I think the pull model is better at least in terms of discovery, and you need less configuration overall for a pull-based monitoring system, as now you need to only configure the monitoring system to know about your services, and the service instances don't need to know: a) who they are, b) where the monitoring system lives. In a push-based world, you need to configure both sides.

I expand on that a bit more in https://prometheus.io/blog/2016/07/23/pull-does-not-scale-or...

Yes, Prometheus is a great choice for that as well. It's pretty easy to write integrations (we call them exporters) to get metrics out of existing third-party systems that you cannot easily instrument directly. Here's a list of exporters we already know about, but it's usually easy to write one of your own if it doesn't exist yet: https://prometheus.io/docs/instrumenting/exporters/

The pulling itself has never been a scaling/bottleneck concern in Prometheus, especially since no individual events are transferred, but just the current state of each metric. The bottleneck is usually the storage's sustained ingestion speed (disk IO or similar).

A single big Prometheus server can easily do millions of time series, and in an older record, 800,000 samples stored per second. You could monitor e.g. 10,000 hosts with that with quite some detail, and the bottleneck is still not the pull aspect.

Mostly, it has great support for service discovery for many major cloud and container platforms (Kubernetes, Marathon, EC2, Azure, Zookeeper, Consul, ...). So it can not only go out and pull metrics data directly from instances as they float around your dynamic cluster scheduler, but also attach identity metadata (as provided by the service discovery) to the time series collected from each instance. For example, you may map EC2 tags or Kubernetes labels into your Prometheus time series labels to give you more useful metrics. There's also a way to plug in your own custom service discovery. Also, Kubernetes exports native Prometheus metrics since quite a while already.

Borg inspired Kubernetes. Borgmon inspired Prometheus. So naturally it works well together with a dynamically scheduled world.

If your monitoring system doesn't have knowledge of what's supposed to be out there (like via service discovery), how do you know when something is broken/down/missing?

I want to understand more about why you think that pull doesn't scale. Borgmon at Google (the biggest scale ever) worked fine with a pull model, and as far as I know, the successor still pulls, too. A push model also has the problem that now the identity of each instance needs to be known by each instance itself, instead of only by the monitoring system via service discovery (which needs to know about what instances should exist and their identity anyways, otherwise how is it going to let you know that something is wrong?).

We also have an FAQ in Prometheus about why we prefer pull: https://prometheus.io/docs/introduction/faq/#why-do-you-pull...?

In my experience, pulling is operationally much nicer than pushing, and I've worked with both. It also gives you somewhat less of an accidental DDoS exposure.

Check out this video around 1h22m: https://youtu.be/6WyaO29XDf8

If you're interested in learning more about the reasoning for the SolarCity acquisition, I can recommend listening to the conference call recording about that: http://edge.media-server.com/m/p/makhvjt8

Ah sorry, he is "brian-brazil" on GitHub.

Our focus is not logging only, it's the opposite: we don't support storing logs of individual events. What Prometheus does is store dimensional numerical time series. See the data model: https://prometheus.io/docs/concepts/data_model/

So it's a question of whether you can squeeze your data into that model and whether you need per-event details or whether aggregated time series are ok.

If the interface you're getting the metrics from is generic and standardized enough to work on most Linux systems, it has a good chance of making it in. "/sys/class/power_supply/..." probably fits that requirement, but bbrazil (same name on GitHub) would be the best to give a final judgement on that kind of thing.

It's still focused on pull, while there is the Pushgateway (see https://prometheus.io/docs/instrumenting/pushing/ and https://prometheus.io/docs/practices/pushing/) for dealing with one-off situations where you cannot scrape something. Pull works great in most situations where people have their own private clouds or datacenters, but it's less suitable for very restrictive environments where you can't run Prometheus on the same network or behind the same firewall as the targets you want to monitor.

In the usual cases, pull has many benefits however:

- You can get high availability by simply running two identically configured, independent Prometheus servers. No clustering required.

- You can run a copy of production monitoring (or similar) on your laptop without changing production. This is great for experimentation and testing changes.

- You get free up-ness monitoring via scrapes and can use this for alerting.

- When there's an HTTP pull endpoint on service instances, you can also go there manually as a human and check out the current metrics state of any target, independent of the Prometheus server.

- Knowledge of service identities is inverted: instead of each service instance having to know its own identity (usually instance="hostname:port" and some job/service name), the monitoring system knows (usually via some form of service discovery) what instances should be there and how they are labeled, and proactively checks on them. Services have no knowledge of where the monitoring system lives anymore, enabling the above use cases.

- Debatable, but push-based monitoring systems can make it easier for someone to accidentally DDoS your monitoring. (still possible with pull, but you have one central place where you know what you pull from)

No business model - Prometheus is 100% free and open-source and independent of any one company. We have also joined the Cloud Native Computing Foundation (https://cncf.io/) as the second member project after Kubernetes: https://cncf.io/news/announcement/2016/05/cloud-native-compu...

Read more here: https://prometheus.io/docs/introduction/overview/ and here: https://prometheus.io/community/

Thanks!

That depends - but yeah, those situations can be tricky sometimes.

If you are pushing metric states regularly though (more than every 5m) or don't set client-side timestamps, that usually works though. But maybe you have an even more special use case there regarding those metrics and the staleness?

Usually you start by some functional sharding (giving each team of services their own Prometheus servers), but also having per-datacenter Prometheus servers and then some hierarchical federation layer ontop of that. There's no built-in automatic horizontal scaling though (which would be going against the design goal of not having a clustered system, for reliability).

Some resources:

- Scaling in general: http://www.robustperception.io/scaling-and-federating-promet...

- Federation: https://prometheus.io/docs/operating/federation/

Not quite, though there is some overlap. Someone else asked me to compare Prometheus to other tools in the APM (application performance monitoring) space, and I'm going to share the summary I came up with:

The way I would describe Prometheus in relation to those other tools:

- Prometheus is open-source and self-hosted.

- Prometheus is about dimensional numeric time series metrics only (no log-based analysis, no per-request tracing, etc.).

- Prometheus has a strong focus on systems and service monitoring, not so much on business metrics.

- Prometheus is more of a Swiss army knife of monitoring rather than a ready-to-drop-in package that starts monitoring everything automatically.

- Prometheus is very much about whitebox monitoring and manually defining any metrics that could be useful for you (although we support blackbox exporting and bridging metrics from existing systems as well).

- We don't do machine-learning-style anomaly detection, but we do alerting based on manually defined rules.

- For a purely metrics-based solution, the insight we deliver is one of the best in the field (via the dimensional data model and the query language to go with it).

- Many open-source projects are starting to expose native Prometheus metrics (like k8s, etcd, ...), which gives Prometheus an advantage when being used together with those.

EDIT: Also try the "Getting Started" tutorial - that should only take a couple of minutes to try it out: https://prometheus.io/docs/introduction/getting_started/

Yeah, it goes from people putting Prometheus on their Raspberry PI all the way to companies like DigitalOcean monitoring millions of machines with Prometheus (https://promcon.io/talks/scaling_to_a_million_machines_with_...).

A lot of open-source projects (Kubernetes, Etcd, ...) are also exposing native Prometheus metrics now, making it easier to integrate with those.

EDIT: also check the PromCon schedule for other user companies giving talks: https://promcon.io/schedule/

And the sponsors are also users: https://promcon.io/#our-sponsors

Prometheus range queries work a bit differently, so this is not 100% reproducible in a graph query, but a similar thing is possible. A given PromQL expression is evaluated at every resolution step along the graph and doesn't have context about what the "graph range" is. At every evaluation point, it can still look back over a given time window, but that's more of a sliding window approach then and independent of what the visible graph time range is.

For example, instead of highestCurrent, you could do something like:

topk(3, my_metric)

This would at every point along the graph select the current top 3 series that have the my_metric metric name.

Or if you want to average each series over the last 10 minutes at every point in the graph before selecting the top 3:

topk(3, avg_over_time(my_metric[10m]))

Note that due to the reasons mentioned above, topk() here does not select whatever line has the largest area under the entire visible graph range, but whatever is at the top at each given resolution step. So you may actually get more than 3 series in your graph, but only 3 at a time at any given X.

There's also an issue asking about this, but we're not sure if that is fundamentally compatible with Prometheus's query execution model without major changes: https://github.com/prometheus/prometheus/issues/586

That would be the correct issue about the staleness limit, yes. Note that Prometheus does not track individual events, but only numerical time series and their current and historical values. What's the exact use case you're stuck on? I guess you are using the pushgateway with client-side timestamps?

Not yet, but one or more companies are working on this as part of their offerings in the future. Stay tuned!

Note that a hosted Prometheus service has different design tradeoffs. As Brian mentioned, Prometheus as it is is really meant to be run as close as possible to your monitored services for maximum reliability. Also, there's no clustered and long-term durable storage for a similar reason, which will likely be different in hosted versions.

Prometheus cofounder here - we're happy to take any questions. Huge congrats to everyone who made this release possible and for all the excellent work over the years that lead up to this!

A couple of us from Prometheus met with Josef from Chronix last week. I haven't tried Chronix yet, but it sounds great for what they're doing. It's currently used for industrial manufacturing monitoring (think car production lines), with a focus on offline anomaly detection. However, there are no live queries yet - it's more about offline number crunching and storing all data points forever. They are also Prometheus users, so they're interested in having Prometheus as a live monitoring system and feeding long-term data into Chronix once the interfaces for that in Prometheus become available.

There is a radar, but it turns off in that situation, as Elon explained in a tweet: https://twitter.com/elonmusk/status/748625979271045121

That's great, but with processes (containerized or not), I can use the full variety of standard diagnostic tools built over many decades. Of course you can start building similar tools for unikernels, but it's gonna take you a long time before you get to a similar state.

Yep, I did. It still doesn't really change my view, especially on the front of operatability/debuggability. If you have ever operated really large services, you know how big of a problem the lack of that is.

It seems this article gleefully admits many of the downsides of unikernels mentioned in https://www.joyent.com/blog/unikernels-are-unfit-for-product..., while being very brief and naive about the upsides (mainly the very contested security argument).

I admittedly haven't studied the whole unikernel space yet, but intuitively they do seem unfit for production unless we spend a decade rebuilding tooling (debuggers, process diagnostics tools, etc.). And even then, other downsides apply, as laid out in the Joyent article.

Happy to change my mind over time if it proves to be the other way around, but for now I'm very skeptical.

In the end, you just need to be able to find enough people who want to go (there are plenty) and who have the money to pay for the trip (Elon hopes to bring down the cost to such a level that someone selling their home on Earth could afford it in many cases - like $500,000 to move to Mars).