Comment by cube2222 | Hacker News Reader

Comment by cube2222 | Hacker News Reader

Parent Full thread

This was as of november.

Raw Prometheus: Isn't able to hold my data.

Thanos: I liked the project, it's architecture and ease of deployment, but after spending a non-trivial amount of time with it I wasn't able to setup any long-term caching. Thanos uses the prometheus storage format. So whenever I was querying one metric, it was downloading all metrics which were in the same block (all metrics basically afaik), this resulted in gigabytes/s of network traffic where it definitely wasn't necessary, and fairly long query times. (I used it with ceph) Though I know the maintainers were planning to add some kind of caching so this may be fixed. By using the native prometheus data format you also don't get storage space savings over it.

Cortex: Didn't spend any time on it, as I expected similar problems as with Thanos, so left it out for the end (which didn't came after all). I know it does contain a caching element.

Victoria Metrics: As far as I know it's very well engineered and performs great. But I see only one active maintainer so am afraid to use it.

M3DB: Requires a non-trivial amount of memory (I have 3 machines, each 128GB RAM to handle 70k writes/s each (though 1 was able to handle 120 and be stable)). However, with all machines on bunches of raid 0 ssd's, querying is quite snappy. You can set it up with different storage resolutions, so you get detailed data for recent queries, but also fast long range queries. It also uses a magnitude of storage space less than raw prometheus. The documentation is lacking in my opinion in terms of performance tuning, however, the code is well written, so I've just spent a while reading it and it exports very good metrics for itself. Network traffic between the m3coordinator (prometheus remote write gateway) and m3db nodes is kinda huge (5-10x the traffic prometheus->gateway) but that wasn't an issue. Another bonus is that it handles statsd metrics, though I haven’t yet tried that.

For anybody afraid of it operationally, I’ve had no problems. It mostly worked as is.

Another comment on m3db:

m3query did have some inconsistencies compared to prometheus in how queries using intervals evaluated. (and sometimes didn't return any data because of it).

Having stepped through the code I don't remember the reason why that was, but I ended up using prometheus instances using remote_read from the m3coordinator (gateway), working like a charm.

Cortex has a design much more like M3DB than Thanos; I wouldn’t expect the same problems as Thanos at all.

(I am a Cortex maintainer)

Thanks for details! Really appreciate it.

> It also uses a magnitude of storage space less than raw prometheus

AFAIK, Prometheus compression is about 1.2-3 bytes per datapoint. A magnitude less is 0.12-0.3 bytes - are these numbers correct?

Here you have the specifics: https://m3db.github.io/m3/m3db/architecture/engine/

I admit I’ve exaggerated a bit as Prometheus doesn’t support downsampling, in m3db I only keep 2 weeks of data at full resolution, 2 months at lower, and 5 years at even lower.

Downsampling might not save as much space as you think depending on how m3db works. https://github.com/thanos-io/thanos/issues/813 goes in to why for a similar project.

Whether it saves space or not, looking at metrics over period of months or years when the data is raw is far more slow/expensive than looking at downsampled data.

If you still want to be able to quickly graph and view old data, downsampling is the only way to keep your queries interactive.

Take for instance 30s data vs 10min data. 20x more computation, network exchange and everything else of that nature needs to happen.

Also if you want to keep only a subset of your data for a very long time, you need to have retention policies - otherwise you end up storing all that extra data forever.

At large numbers (terabytes to petabytes) this stuff is impactful, at smaller numbers (gigabytes) my points here are far less relevant.

I know about the Thanos problem, and it's one of the things I didn't really like about it using the prometheus storage format.

It does work in m3db.

Following are the on-disk sizes of one replica:

2 weeks at 15s res: 90G

2 months at 1m res: 160G

3 months at 5m res: 60G

EDIT: The only thing is that m3db doesn't really downsample. You just create one namespace (table) for each resolution, and set the m3coordinator up so that it writes to each at the wanted interval, then set up a different retention for it. (this way you have duplicates in recent data)

M3 namespaces aren't set up for a specific resolution. The writer decides and you can write various series in different resolutions to one namespace in theory.

> Victoria Metrics: As far as I know it's very well engineered and performs great

As someone who has auditioned it, briefly, let me assure you that it is certainly not the former, and only appears to be the latter due to a lot of cut corners and spec-violating implementations.

Could you elaborate on the specifics? I'd be very interested.

Reply on news.ycombinator.com