I wonder what InfluxDB means by "distributed", that is, if I could use it to implement a push (where distributed agents push to a centralized metric server) model.
I wonder what InfluxDB means by "distributed", that is, if I could use it to implement a push (where distributed agents push to a centralized metric server) model.
I wouldn't totally rule out the pushgateway for this use case. If we decided to implement a metrics timeout in the pushgateway (https://github.com/prometheus/pushgateway/issues/19), this would also take care of stale metrics from targets that are down or decommissioned. The pushing clients would probably even want to set a client-side timestamp in that case, as they are expected to be pushing regularly enough for the timestamp to not become stale (currently Prometheus considers time series stale that are >5 minutes old by default, see also the "Improved staleness handling" item in http://prometheus.io/docs/introduction/roadmap/.
For metrics I go Riemann->Graphite. Riemann comes with a graphite compatible server so I push straight to that for processing and alerting. I also send from Riemann to logstash and logstash to Riemann where it makes sense.
For my metrics dashboard I use Grafana which is really awesome. I make use of its templating pretty heavily as I run a lot of individual IIS clusters. I can create cluster overview and drill down dashboards and template it so I can just change the cluster number to see its stats. You can also link directly into a dashboard passing variables as a query string parameter. Pretty excellent.
One challenge with that model of metrics is that it assumes that the monitoring system doesn't have much ability to work with samples, so it makes up for this by calculating rates and quantiles in the client. This isn't as accurate as doing it in the server, and some instrumentation systems don't allow you to extract out the raw data to allow you to do it in the server.
For example say you've a 1-minute rate exported/scraped every minute. If you miss a push/scrape you lose all information about that minute. Similarly if you're delayed by a second, you'll miss a spike just after the previous push/scrape. If instead you expose a monotonically increasing counter and calculate a rate over that in the server, you don't lose that data.
My full thoughts are up at http://www.boxever.com/push-vs-pull-for-monitoring The short version is that I consider pull slightly better, but not majorly so. Push is more difficult to work with as you scale up, but it is doable.
Yeah, I did that benchmark with 11x overhead for storing typical Prometheus metrics in InfluxDB in March of 2014. Not sure if anything has changed conceptually since then, but if anyone can point out any flaws in my reasoning, that'd be interesting:
https://docs.google.com/document/d/1OgnI7YBCT_Ub9Em39dEfx9Bu...
We can't pull them, because hitting the load balancer would randomly choose only one instance.
Instances are scaled up based on load. So we can't specify the target instances in Prometheus because it keeps on changing.
We'd like to try this out, but any ideas what to do for the above?
We're working on service discovery support[1] so that you can dynamically change what hosts/ports Prometheus scrapes. Currently you can use DNS for service discovery, or change the config file and restart prometheus.
Previously, you would have had to encode metadata in the series name. Otherwise if you used string columns you'd see a massive waste in disk space since they were repeated on every measurement.
In the next version of InfluxDB (0.9.0), you can encode metadata as tags and it gets converted to a single id.
With either of those schemes you should see much better numbers on storage.
My understanding is that InfluxDB will work for your use case, as it's all push.
> I see the pushgateway, but it seems deliberately not a centralized storage.
Yeah, the primary use case for the pushgateway is service-level metrics for batch jobs.
May I ask why accessing your producers is a problem? I know for Storm I was considering writing a plugin that'd hook into it's metrics system, that'd push to the pushgateway and Prometheus would then scrape. Not my preferred way of doing things, but some distributed processing systems need that approach when work assigment is dynamic and opaque across slaves.
A pull model, while technically distributed, is organizationally centralized: I have to get each node's owner to grant me direct access. Politically, and for security reasons, that's not going to happen.
http://www.boxever.com/push-vs-pull-for-monitoring looks at other bits of the push vs. pull question.
From my experience with distributed systems push replication will get you in trouble, very soon.
Edit: I was too quick to post this question. An obvious scenario is where the client is behind a firewall. Never mind me. I am an idiot.