Hacker News Reader

Comment by ibspoof | Hacker News Reader

When my team agreed to use Prometheus from the client side we looked at Thanos, Cortex, and M3DB, but none of them gave us the flexibility and comfort of adoption for a small team providing a service to 10s of internal groups. We have many private internal DCs and needed metrics to be stored in the cloud, pulling data to the cloud seemed awkward and required access rights we couldn't get.

We ended up using Postgres 10 w/ TimeScaleDB and their Prometheus plugin with a simple emulated push gateway that converts a prom formatted http post to a postgres batch insert. Postgres is 3 nodes monitored with Patroni.

Working great for us and handling 1000+ metrics a second with ease and we get SQL for both real-time metrics for monitoring and analytics for business needs. We are using about 10-15% of our systems giving us room to grow.

View on HN

> We have many private internal DCs and needed metrics to be stored in the cloud, pulling data to the cloud seemed awkward and required access rights we couldn't get.

You mean having a Prometheus server run in the cloud, but then pulling from on-prem things from the cloud? Not sure how either Cortex or Thanos would require that, as you'd still run on-prem Prometheus servers for them, but then collected data is pushed to the cloud in the end. But maybe I'm misunderstanding what you mean here.

> Working great for us and handling 1000+ metrics a second

Curious about this - I would expect any system to be able to do that easily, as that's a tiny, tiny amount. A single big Prometheus server can do roughly 1000x that (I think someone once managed to do 1M samples/second ingested).

TimescaleDB co-founder here. Glad to hear it is working well for you. If you have any feedback on how we can make it work even better, would love to here it - ajay (at) timescale.com

For others reading this: TimescaleDB can easily scale to millions of metrics a second [0][1], but to be honest we've found that few folks need that level of scale.

But people do care about whether or not they can trust something new. If you're already comfortable with Postgres then you're already comfortable with TimescaleDB.

Also! Other solutions are great too. E.g., I really like the Thanos model with object storage. I don't think there is a "best" option out there - it really just depends on what you want to optimize for your use case.

[0] https://blog.timescale.com/blog/building-a-distributed-time-...

[1] https://news.ycombinator.com/item?id=20760324

Ingestion rate of 1K metrics per second can be handled by any solution on low-end hardware. I believe even ancient Raspberry PI would feel comfortable with such a load :)

VictoriaMetrics scales to 50M+ metrics per second on a single server [1], [2].

[1] https://mobile.twitter.com/MetricsVictoria/status/1209116702...

[2] https://mobile.twitter.com/MetricsVictoria/status/1209186575...

Reply on news.ycombinator.com