jrv

> Though the only real new Bank in the UK to do well, traction wise would be Metro Bank

I'm not in the UK, but isn't Monzo also doing quite well?

No, sorry, it was a private commercial training.

There's different tradeoffs... Prometheus's own federation is a pretty simple scrape-time federation - a Prometheus server pulls over the most recent samples of a subset of another Prometheus server's metrics on an ongoing basis. Thanos does query-time federation rather than actually collecting and persisting data for all "federated" servers in a central place (other than the e.g. S3 bucket for long-term data). So with Prometheus federation you have to choose pretty carefully which aggregated stats you'd like to pull into some higher-up Prometheus layer, and then you only have access to those (in that server). Thanos allows you to query over the data in multiple Prometheus servers at once, in all their detail.

And I think Cortex is mostly useful for people who want to run a big centralized, multi-tenant service in their org to keep all the global view and long-term data. (most people tend to use Thanos)

> We have many private internal DCs and needed metrics to be stored in the cloud, pulling data to the cloud seemed awkward and required access rights we couldn't get.

You mean having a Prometheus server run in the cloud, but then pulling from on-prem things from the cloud? Not sure how either Cortex or Thanos would require that, as you'd still run on-prem Prometheus servers for them, but then collected data is pushed to the cloud in the end. But maybe I'm misunderstanding what you mean here.

> Working great for us and handling 1000+ metrics a second

Curious about this - I would expect any system to be able to do that easily, as that's a tiny, tiny amount. A single big Prometheus server can do roughly 1000x that (I think someone once managed to do 1M samples/second ingested).

Prometheus author here (Julius).

Generally agreed that you can get far with a single Prometheus server (or many independent vanilla Prometheus servers, potentially also using Prometheus's own federation). But I still recommend Thanos as an extension to a lot of people. I like Thanos because it's so easy to deploy alongside an existing Prometheus installation, while itself being mostly stateless (long-term state is kept in object storage), and it gives people:

- a global view over multiple Prometheus servers - deduplicated view over servers in an HA pair - durable long-term storage for little cost

The Thanos architecture diagrams (especially the one in their README.md) can look a bit intimidating, but I find it sooo easy to get started with in practice, since you don't even need to deploy all of the components to begin with it. I usually tell people to just drop in a Thanos sidecar next to each of their Prometheus servers, so they will get backups of all their Prometheus server data (for those who are interested in long-term data retention). And then later, they can add the Querier component for an integrated view over multiple servers. And then later they can deploy the Store gateway to also integrate back long-term data into that view. And then at some point, the compactor...

Without being a Thanos expert, it took me ~15 minutes to deploy all those components (+ Minio for object storage) in front of a training audience that wanted to know more about Thanos (while reading Thanos + Minio docs). Of course a proper production deployment always takes way more time, but still I like how conceptually simple it is to integrate Thanos with Prometheus.

You continue to claim that VictoriaMetrics is backwards-compatible with Prometheus and only include a side-note that it actually is not in all cases. Such knowingly misleading statements are dangerous for users and the ecosystem.

As for the more general point, my comment at https://news.ycombinator.com/item?id=21772791 stands. It's not about competition (we welcome and love that from all sides), it's about overly marketing a product on Prometheus OSS community channels, while at the same time not being clear about or ignoring compatibility issues that will for sure cause problems for the ecosystem.

EDIT: One more point: The people seeing a problem around VM behavior here are not just Prometheus team members. Multiple external community members / users have pointed this out as well to us.

I'm really interested in the topic and would love to just lurk and read what people say on there for a while, but the text in the signup form makes it sound like a person like me wouldn't be accepted, since I don't have super concrete plans to make that my next profession (although I'm generally interested in exploring this field if I really started thinking that my skills would make sense there).

First approaches are happening for this now:

https://groups.google.com/forum/#!topic/prometheus-users/wO5...

- https://github.com/Percona-Lab/prom-migrate (requires old Prom server to run for reading out data)

- https://github.com/juliusv/prom-data-migrator (operates offline on old and new storage dirs directly)

The question is about tracking and storing individual events (logs) with arbitrary per-item detail vs. dimensionally limited aggregations (time series / metrics). In either case, I think we agree that the data should be recorded in a structured way, and when I say "logs" I just mean a record of individual items, not of sampled/aggregated metrics.

Given that, you need both logs (individual events) and metrics. Logs give you crucial insight into individual interesting events such as single requests that bring your service down, but logs are orders of magnitude more expensive than metrics in tracking, storage, and processing. So that's why you use metrics for a much wider scope and for longer time periods.

The problem is (as you know) that single machines are in practice still too reliable and Prometheus is still too good at storing data for long times that many people have come to rely on it despite warnings :)

Out of curiosity, do you care about migrating the data online, or would a brief Prometheus downtime (and thus gap in data) be ok?

I am not too familiar with the PG-TimescaleDB adapter, but the storage engine rewrite in Prometheus was definitely the most important improvement in Prometheus 2.0.

There are other changes than the storage though that are outlined in the migration guide: https://prometheus.io/docs/prometheus/latest/migration/

* Minor PromQL changes (mainly removals of deprecated features)

* Staleness handling (not sure if the TimescaleDB storage handles this correctly, as it relies on storing a special NaN value bit-wise correctly for marking series stale)

* New rule file format is now YAML and supports per-group evaluation intervals and serialization of rule executions within one group (in case of data dependencies)

* Other minor bits

Yeah, as Prometheus's local storage is meant more as a transient / non-durable metrics store, the only current way to migrate while simultaneously accessing old and new data is to run both the old and new servers and have the new one read old data from the other one via the remote-read integration.

Someone could write a tool to do a full migration of the old storage format to the new one, but the formats are completely different and at least in the naive version of such tooling, that would have to happen offline and take a very long time to run for large storages.

EDIT: If you would like to fund development of such a tool, let us know :)

No, we never used flot charts. We're using Rickshaw since the beginning: http://code.shutterstock.com/rickshaw/

It was unmaintained for a while, but now it seems there've been commits in the recent past again. However, for real dashboarding (not just simple ad-hoc queries), we recommend using Grafana anyway.

Nice to see that this comes with Prometheus and OpenTracing instrumentation!

It seems like it. Envoy was already competing with Linkerd, and "Istio uses an extended version of the Envoy proxy" (source: https://istio.io/docs/concepts/what-is-istio/overview.html). So it does that and more.

Yeah, and the global imagery is also not real time (so you can't track a car's journey, as the article claims), but daily. It's a BS article.

Great. Signal doesn't do group calls though, right? Is there any app that does end-to-end-encrypted group calls?

If you read about this all the time, then you're probably also familiar with the Fermi Paradox and the possible explanations for it. 2016 may have given us a clue as to which one of them seems most likely.

By a couple of years, but not many decades, yes.

Considering that Elon Musk says we'll have full self-diving cars (from an engineering standpoint) in 2 or 3 years, this estimate seems very pessimistic.

Since GitLab is heavily betting on Prometheus now, my guess is that it will be used to cover almost all of those points and more. Yep, lots of work though.

Disclaimer: Prometheus co-founder and just started as a Prometheus contractor for GitLab.

EDIT: ah, didn't reload the page to see that sytse had already responded with this :)

Is there any hosted monitoring solution that integrates with service discovery, so that it's actually useful for serious alerting in nowadays' dynamic environments? Otherwise you can't even tell if things that should be there are reporting in or missing.

See http://blog.runnable.com/post/153498635761/how-we-saved-98-o... as a counter point.

Case in point: http://blog.runnable.com/post/153498635761/how-we-saved-98-o...

An argument why I believe consciousness can affect particles/"physics": if it couldn't, why would we be speaking about our internal experiences? If they were only experiences like in a movie, the fact that they exist wouldn't influence anything and our bodies wouldn't talk about them. Yet here I am, typing about internal experiences.

That, plus they said that you'll often want to use the car at the same time as someone else uses electricity in the house.

As per the earnings call, the profit per unit actually went up, and the discounts were never sanctioned and also not relevant to this end result.

Exactly! There were very few discounts, and they were never sanctioned to begin with. Elon made it clear in that internal email that these were not allowed by Tesla's internal policy. Had nothing to do with this quarter's profitability, as reaffirmed by him in today's earnings call.

E2E is already activated in Riot web and will come to mobile in a month or so (at least according to what Matthew said in the Matrix room on Matrix).