Prometheus, but Bigger | Hacker News Reader

Prometheus, but Bigger

luizrojo.medium.com

57 pointskiyanwang15 comments

The key constraint here is that the author has no access to persistent disks and can only use object storage for persistence. Otherwise Thanos would be extreme overkill for this number of metrics.

Single-node VictoriaMetrics can easily handle 1M metrics/sec

> Thanos would be extreme overkill for this number of metrics

Data volume is just one thing. Thanos makes Prometheus stateless and easy to shard, all in a non-invasive approach that is solid, boring, and just works. The architecture works well in small scale systems. I even use it in a single node k8s cluster in my homelab, pays only about ~$1 a month for Backblaze B2 so I never worry about data retention or disk usage.

> Single-node VictoriaMetrics can easily handle 1M metrics/sec

Even if I have disk access, I would think twice before deploying a database and manage it myself when I don't have to. Besides the maintenance burden and potential scaling issues in the future, it may cost you more to use block storage like EBS than S3.

Also, Prometheus memory usage overhead for remote write was wild[1], so, good luck with capacity planning and config tweaking.

1. https://prometheus.io/docs/practices/remote_write/

The solution here is meant to be expanded easily and to keep things at a low cost.

The “insane” prices were from NewRelic, using the Kubernetes integration and infrastructure monitoring. I am based in Brasil, so every price is “times 5” plus taxes.

Right now we have a small number of clusters, but it will be growing on a fast pace in the future. For us, monitoring the new clusters will come easy as deploying a couple of new prometheus PODs inside the new cluster, adding a couple lines to our deploy pipeline.

As for doing in house, since we have a complete private cloud solution, it makes sense for us, as mentioned by zzyzxd in another comment, Thanos is pretty straight forward and just works, it demands almost zero maintenance, we just keep an eye for storage consumption and new releases.

I dont understand the need to NiH things, why not clickhouse with weekly, monthly, etc aggregation. It comes with built in sharding, can store time series data somewhat efficiently and doesn't have some hack h/a setup (in general not thanos).

Wouldn’t Clickhouse be the NIH solution? Thanos supports Prometheus natively, but Clickhouse doesn’t (or at least doesn’t appear on the integrations list in the Prometheus docs), so you’d have to write an adapter for it.

I'll bite, no because Prometheus, Thanos, etc all have to reinvent sharding (Thanos), a query language (Promql), vectorizing (Prometgeus can't), transactions (Prometheus internals), etc when it all exists in clickhouse and uses somewhat standard SQL. There are trade-offs but imo it's not worth NiHing something

> For the monthly expenses, with most of the components running on-premises, there was a 90.61% cost reduction, going from US$ 38,421.25 monthly to US$ 3,608.99, including the AWS services cost.

I might be missing something, but $3k/month (let alone $38k/month) sounds absolutely insane to me for how little metrics they're collecting (4k5 metrics per second, 2.7TB of data per year). Is the money going for network bandwidth or something along those lines?

AWS, for example, charges for cross-AZ data transfers. Naive setups with multiple AZs (us-west-1a, 1b, etc.) and a centralized Prometheus setup will rack up quite the cost.

The reducing cross-AZ data transfer savings on one service resulted in a low 6 figure per year savings. Its something we overlooked during initial setup and now its something I check for when dealing with AWS networking.

Out of curiosity, what's your plan for recovery during an AZ or regional outage?

If RDS or other “hard to replicate very quickly during disaster” infra is being run I personally would still have cross A-Z replication at minimum, to reduce network costs I would configure the “other zone” as a backup replica only and not for performance clustering.

With automation we can spin up full new compute stacks, including load balancers and DNS in about 5-10 minutes per “unique” environment configuration.

While it guarantees we could never have a no downtime failover, we’re okay with it and have more than halved our network costs (which admittedly were about number 8 on our AWS bill by cost).

If AWS has a regional outage this service is so far down the list of services to recover/restore that it probably will be overlooked. Accept the increased risk for the cost savings since it meets the reliability requirements of the service.

They say they ingest 226gb per month. The cross-az transfer cost is $0.01 per gb. So that should come out to only $2.26/month for them.

They are also storing the data in S3 (ingest bandwidth, storage). Plus running an RDS instance (instance cost, x2 if replicated, bandwidth again possibly x2 or more, IO cost) as well as local storage (EBS size/IO cost). And the size of the raw metrics might not be the wire size, especially if it’s uncompressed JSON. My guess on reviewing the post is that a lot of their savings were inter-AZ bandwidth conservation. But it’s hard to say without poking around in their AWS console :)

I agree. I'm working with a metrics system that takes just over 1 million metrics a second and it has a similar run rate to ~38k a month.

Reply on news.ycombinator.com