Are Prometheus and Thanos developers against fair competition? | Hacker News Reader

Are Prometheus and Thanos developers against fair competition?

8 pointsvalyala17 comments

Hi all.

I'm the core developer of VictoriaMetrics - open source time series database, which can be used as remote storage for Prometheus monitoring system. It implements opinionated query language, which is backwards compatible with PromQL from Prometheus. Is it called Extended PromQL [1]. It slightly derives from the original PromQL in controversial cases [2]. Additionally it implements user-friendly functionality on top of PromQL:

* `WITH templates` (aka `CTE expressions`) [3].

* User-friendly simplifications such as `rate(q)` as a shorthand to `rate(q[$__interval])`.

* Additional functions and operators aimed towards solving practical issues, which cannot be solved with the original PromQL.

VictoriaMetrics constantly evolves and sometimes implements Prometheus feature requests. For instance, recently it gained the ability to show the lower and upper bounds of the estimation returned by `histogram_quantile()` [4]. I mentioned about this in the original feature request, so users could find the solution. This led to the post from Prometheus devs [5], where they:

* Blame VictoriaMetrics for pointing Prometheus and Thanos users to solutions for their issues in public channels.

* Say that VictoriaMetrics cannot use `PromQL` word in the documentation, since it is registered trademark. Why anybody didn't patent SQL yet?

To me this looks like a case for unfair competition. What do you think about this situation?

[1] https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/ExtendedPromQL

[2] https://github.com/prometheus/prometheus/issues/3746

[3] https://victoriametrics.com/promql/expand-with-exprs

[4] https://github.com/prometheus/prometheus/issues/5706

[5] https://github.com/prometheus/prometheus/issues/5706#issuecomment-564532929

Yet another case of "fair" competition from Prometheus and Thanos devs [1]:

* Prometheus users proposed extending `/api/v1/label/.../values` API [2], which has been already implemented in VictoriaMetrics.

* I mentioned about this and described high-level architecture for this feature, so it can be implemented by Prometheus devs.

* They say "you cannot point Prometheus users to the solution in VictoriaMetrics, since this is self-promotion" [1].

The same issue already contains Thanos promotion in the first comments, but nobody complains about this.

[1] https://github.com/prometheus/prometheus/issues/6178#issueco...

[2] https://github.com/prometheus/prometheus/issues/6178

I took offence with @valyala posting about Victoria Metrics in the Cortex Slack channel in the past, but I'm so glad he did because we had _endless_ issues with Cortex and we found Victoria Metrics to be _incredibly_ fast and reliable after trying it when he suggested that it could do something Cortex couldn't.

This github response wasn't cross promotion though -- this was @vayala explaining how it could be implemented, as someone who has done it. The fact that no one complained when Thanos was mentioned, but complained when VM was mentioned shows how hypocritical some Prometheus contributors are.

Honestly, @valyala should just create a metrics collector and then I wouldn't have to deal with the "we know better" attitude from some Prometheus contributors.

The Victoria Metrics contributors are absolutely amazing. Every issue or suggestion has been implemented within days (sometimes hours) and every question answered in incredible detail, with warmth and friendliness.

Thanks for the warm response!

> Honestly, @valyala should just create a metrics collector and then I wouldn't have to deal with the "we know better" attitude from some Prometheus contributors

We didn't want going this route, since this could hurt Prometheus community in the long run. But we constantly keep hearing this feature request from VictoriaMetrics users, who deal with high ingestion rates. Probably such a collector will be implemented in the future unless Prometheus devs address the following issues in Prometheus:

* Increased RAM and CPU usage when enabling remote_write.

* Increased disk space usage on high ingestion rate.

* High network bandwidth usage for pushing data to remote storage.

Sometimes when I read your comments at Prometheus issues or at twitter, it sounds like you want to put prometheus into bad light. It often sounds like you're subliminally insulting prometheus.

Would you mind to share few particular cases? I'm wondering if that's a matter of your personal perception or something more general.

I'm trying hard to write polite comments, which could help users finding the solution for their issues.

My comments could be harsh only in response to misleading posts about VictoriaMetrics [1].

[1] https://www.robustperception.io/evaluating-performance-and-c...

You continue to claim that VictoriaMetrics is backwards-compatible with Prometheus and only include a side-note that it actually is not in all cases. Such knowingly misleading statements are dangerous for users and the ecosystem.

As for the more general point, my comment at https://news.ycombinator.com/item?id=21772791 stands. It's not about competition (we welcome and love that from all sides), it's about overly marketing a product on Prometheus OSS community channels, while at the same time not being clear about or ignoring compatibility issues that will for sure cause problems for the ecosystem.

EDIT: One more point: The people seeing a problem around VM behavior here are not just Prometheus team members. Multiple external community members / users have pointed this out as well to us.

> You continue to claim that VictoriaMetrics is backwards-compatible with Prometheus and only include a side-note that it actually is not in all cases

* There are a few controversial cases where VictoriaMetrics' implementation for PromQL intentionally diverges from the original implementation in Prometheus. This discrepancy is needed because VictoriaMetrics tries hard to make PromQL more user-friendly at edge cases, which aren't solved for a long time on Prometheus side [1].

* Of course, there may be non-intentional discrepancies in PromQL implementation for VictoriaMetrics, because it is implemented from scratch without using Prometheus data model and source code. Such discrepancies are usually fixed as soon as users report about them.

> It's not about competition (we welcome and love that from all sides)

I see. Facts mentioned in this topic shows the real love :)

[1] https://github.com/prometheus/prometheus/issues/3746

What more than a "side note" do you expect when someone claims backwards compatibility?

I deployed Victoria Metrics into my environment and knew _exactly_ which parts deviated from Prometheus' PromQL implementation because it's in the docs.

Some of the core Prometheus maintainers come across as arrogant and stubborn, which definitely hurts the ecosystem.

This is my response to ongoing discussion at https://github.com/prometheus/prometheus/issues/6178 because github issues is truly the wrong place for doing this.

>> @hagen1778 My comment at #5706 (comment) goes into this a bit more. It can indeed be helpful to cross-link to other products/projects at times, and I don't see this individual comment as a big problem - but it fits into an overall pattern that is less ok (see details in my linked comment).

@juliusv I am familiar with the comment you linked. I am also sharing your opinion about being `a bit more sensitive about using Prometheus channels`. However, I find it a bit unfair comparing to other OSS mentions in Prometheus issues. Anyone can verify this easily by searching for `thanos`, `cortex` or `victoriametrics` on https://github.com/prometheus/prometheus/issues

I disagree with second point about `own implementation of PromQL`. The reason why it was re-implemented(AFAIK) with so much efforts instead of just reusing the existing Prometheus code is the unwillingness to extend existing implementation. I'm pretty sure you are familiar with that, because this discussion appeared in your own proposal ticket from Jun 25. And I understand this, authors should protect their projects from being overwhelmed by other people proposals and needs to keep it solid. But I don't understand why it is not allowed to mention the alternative way for users on public platforms like github.

I am very grateful for Prometheus and the ecosystem around it and strongly believe it becomes a standard for the industry, so we all should work on it. That's why such confrontations make me sad. I hope that such situations will never occur in future and opensource will remain the place for creating cool software.

To be fair, members of Thanos and Cortex projects do contribute code to Prometheus, which might explain why they are named more often here.

To be fair, VictoriaMetrics also contributes to Prometheus ecosystem by providing open source cost-effective remote storage for Prometheus, which is easy to setup and operate.

Additional contributions to Prometheus ecosystem include:

* Filing issues and feature requests for Prometheus [1], [2], [3].

* Publishing introduction materials for Prometheus users [4], [5].

* Helping Prometheus users at prometheus-users Google group [6].

[1] https://github.com/prometheus/prometheus/issues/4456

[2] https://github.com/prometheus/prometheus/issues/4769

[3] https://github.com/prometheus/prometheus/issues/5166

[4] https://medium.com/@valyala/promql-tutorial-for-beginners-9a...

[5] https://medium.com/@valyala/prometheus-storage-technical-ter...

[6] https://groups.google.com/forum/#!searchin/prometheus-users/...

I explicitely said "code".

Thanos and Cortex use some code from prometheus. That can explain that. Try looking timescale or opentsdb in the issues/pr.

An additional case of unfair competition was at PromCon 2019. There was a talk named `Remote storage wars` from Adidas at the conference [1]. Various remote storage solutions for Prometheus were compared in the talk for use cases specific to Adidas - Cortex, M3DB, VictoriaMetrics and Thanos receiver. VictoriaMetrics showed the best performance, the lowest RAM usage and the lowest disk space usage.

The was a video recording, but unfortunately slides were broken during the talk [2]. I was hoping that slides will be published in the next months after the conference. Unfortunately slides for this talk weren't published yet, while slides from other talks are already published - see `slides` link at the bottom of every talk page from [3].

This doesn't look very fair :)

[1] https://promcon.io/2019-munich/talks/remote-write-storage-wa...

[2] https://youtu.be/4vZ1PqZ_Foc?t=16025

[3] https://promcon.io/2019-munich/schedule/

The videos will be uploaded soon. Don't worry, none of the slides or other content was lost.

We're just waiting for our video company to finish the final renders with all the correct color and lighting quality adjustments made. Each talk will be individually published on YouTube just like all previous PromCon events.

Great news! I hope this isn't a sarcasm :)

Another case of unfair competition is the article [1] published by Prometheus author, where he claims VictoriaMetrics doesn't compress random data well. I hope Prometheus author knows information theory [2] and was aware that random data cannot be compressed. Then it is unclear why he decided to publish the comparison :) I published response to this article [3], but unfortunately it isn't referenced from the original article spreading the FUD.

[1] https://www.robustperception.io/evaluating-performance-and-c...

[2] https://en.wikipedia.org/wiki/Information_theory#Entropy_of_...

[3] https://medium.com/@valyala/evaluating-performance-and-corre...

Reply on news.ycombinator.com