Hacker News Reader

Comment by x87678r | Hacker News Reader

I love prometheus. Its simple and the built in charts are enough without having to use Grafana on top.

The question is how to do long term storage though. Something I've had a bit of trouble rationing about.

Right now all of my metrics are sitting in a PVC with a 30d retention period, so we're probably fine but for longer term cold storage the options aren't great unless you want to run a custom Postgres instance with the Timescale plugin or something else more managed.

I think that's what https://thanos.io/ is for.

Thanks for the link, I'll check it out.

Do you really need long term? I hate throwing away data but realistically I never really need old performance data. Some stats data is worth keeping but you can extract a few important time series and store them elsewhere.

In any case, Prometheus is throwing away data if a scrap can’t be done. As clearly described on the website, prometheus is not a metrics system. So influx and Prometheus are quite different.

Mistake in my previous message. I wanted to say that Prometheus is not a log system. Metrics could be lost in scraping issues. This is ok in some cases but you have to know that you can loose data.

I second the Thanos link, but if you want to only use Prometheus, I would look into federation:

https://www.robustperception.io/scaling-and-federating-prome...

https://prometheus.io/docs/prometheus/latest/federation/

You can set up a new "cold storage" Prometheus with a longer retention that scrapes select metrics from your regular 30d Prometheus to store for longer periods of time.

How about another Prometheus server in federation[0]?

It's as simple as another Prom instance pointing at the "live" Prom instance. You can filter only the metrics you want to keep for a long amount of time, and downsample if necessary (by just setting a higher scrape interval on the "long term" Prom).

Since this "long term" Prom isn't in the critical path you could skimp on processing resources and just give it a big disk as a cost optimization.

If you are on AWS (or equivalent) the storage there is pretty durable. On-prem you can run HA (two instances with identical config).

[0] https://prometheus.io/docs/prometheus/latest/federation/

For long term retention, look at VictoriaMetrics

My vote is for VictoriaMetrics - the best option if you want to keep things simple and get great performance in the same time!

It's sad to see technically inferior products having more popularity.

VictoriaMetrics pretty much has everything you'd hope for as a Prometheus long term storage, like direct PromQL support, good performance and ease of installation.

Thanos or Cortex

I don't. It's quite difficult to do complex queries and its query language has a few gotcha. I think it's quite good and one of the best solutions today, but I look forward to something as simple and as fast, but with a proper query language.

Yes would be nice to be able to run sql or something more standard. I like that its just simple, one application and it just works.

PromQL from Prometheus has steep learning curve because of lack of good documentation. When you understand PromQL basics, then it is much easier to write typical queries over time series data in PromQL than in any other query languages (SQL, Flux, InfluxQL, etc.). I'd recommend reading the article about PromQL basics [1] and then feel the power of PromQL.

[1] https://medium.com/@valyala/promql-tutorial-for-beginners-9a...

Reply on news.ycombinator.com