Having deployed m3 recently, I’ve not found an alternative which is cost effective and fast at the same time. Granted, it uses a lot of memory, but other than that I’ve been incredibly happy with it.
Raw Prometheus: Isn't able to hold my data.
Thanos: I liked the project, it's architecture and ease of deployment, but after spending a non-trivial amount of time with it I wasn't able to setup any long-term caching. Thanos uses the prometheus storage format. So whenever I was querying one metric, it was downloading all metrics which were in the same block (all metrics basically afaik), this resulted in gigabytes/s of network traffic where it definitely wasn't necessary, and fairly long query times. (I used it with ceph) Though I know the maintainers were planning to add some kind of caching so this may be fixed. By using the native prometheus data format you also don't get storage space savings over it.
Cortex: Didn't spend any time on it, as I expected similar problems as with Thanos, so left it out for the end (which didn't came after all). I know it does contain a caching element.
Victoria Metrics: As far as I know it's very well engineered and performs great. But I see only one active maintainer so am afraid to use it.
M3DB: Requires a non-trivial amount of memory (I have 3 machines, each 128GB RAM to handle 70k writes/s each (though 1 was able to handle 120 and be stable)). However, with all machines on bunches of raid 0 ssd's, querying is quite snappy. You can set it up with different storage resolutions, so you get detailed data for recent queries, but also fast long range queries. It also uses a magnitude of storage space less than raw prometheus. The documentation is lacking in my opinion in terms of performance tuning, however, the code is well written, so I've just spent a while reading it and it exports very good metrics for itself. Network traffic between the m3coordinator (prometheus remote write gateway) and m3db nodes is kinda huge (5-10x the traffic prometheus->gateway) but that wasn't an issue. Another bonus is that it handles statsd metrics, though I haven’t yet tried that.
For anybody afraid of it operationally, I’ve had no problems. It mostly worked as is.
> It also uses a magnitude of storage space less than raw prometheus
AFAIK, Prometheus compression is about 1.2-3 bytes per datapoint. A magnitude less is 0.12-0.3 bytes - are these numbers correct?
I admit I’ve exaggerated a bit as Prometheus doesn’t support downsampling, in m3db I only keep 2 weeks of data at full resolution, 2 months at lower, and 5 years at even lower.
If you still want to be able to quickly graph and view old data, downsampling is the only way to keep your queries interactive.
Take for instance 30s data vs 10min data. 20x more computation, network exchange and everything else of that nature needs to happen.
Also if you want to keep only a subset of your data for a very long time, you need to have retention policies - otherwise you end up storing all that extra data forever.
At large numbers (terabytes to petabytes) this stuff is impactful, at smaller numbers (gigabytes) my points here are far less relevant.
It does work in m3db.
Following are the on-disk sizes of one replica:
2 weeks at 15s res: 90G
2 months at 1m res: 160G
3 months at 5m res: 60G
EDIT: The only thing is that m3db doesn't really downsample. You just create one namespace (table) for each resolution, and set the m3coordinator up so that it writes to each at the wanted interval, then set up a different retention for it. (this way you have duplicates in recent data)
M3 namespaces aren't set up for a specific resolution. The writer decides and you can write various series in different resolutions to one namespace in theory.
As someone who has auditioned it, briefly, let me assure you that it is certainly not the former, and only appears to be the latter due to a lot of cut corners and spec-violating implementations.
m3query did have some inconsistencies compared to prometheus in how queries using intervals evaluated. (and sometimes didn't return any data because of it).
Having stepped through the code I don't remember the reason why that was, but I ended up using prometheus instances using remote_read from the m3coordinator (gateway), working like a charm.
(I am a Cortex maintainer)