roskilli

I think ScyllaDB would have definitely done better than Cassandra (which we were using alongside ElasticSearch), although another thing I mention in this thread is that a lot of existing distributed databases do not have a multi-dimensional inverted index available that can index keys in there primary storage engine.

This makes it tough to use solely either ScyllaDB, ClickHouse or Cassandra for that matter for metrics workloads at scale since they need to find a needle in a haystack - a few thousand time series amongst a set of millions to billions, where users only specify a subset of the dimensions on the metrics in any order they want to. This is hard to do without an inverted index.

Yes I've seen that also work, it's a lot of stitching together things yourself and we had to put a lot of caching in front of the inverted index we were using, however definitely plausible. ClickHouse doesn't do any streaming of data between nodes as you scale up and down which was a big thing for us since we had large datasets and needed to rebalance when cluster expanded/shrunk.

With regards to trie vs inverted index for Graphite data, I'd actually still be inclined to say inverted index is better based on the amount of queries I saw at Uber with Graphite where people did `servers.*.disk.bytes-used` type queries which is way faster to do using an inverted index since you have a postings list for each part of the dot-separated metric name, rather than traversing a trie with thousands to tens of thousands of entries in index 1 host part of the Graphite name. This is what M3DB does[0].

[0]: https://github.com/m3db/m3/blob/b2f5b55e8313eb48f023e08f6d53...

I want to first say, I have a great amount of respect for Netflix's engineering and for Atlas itself, it's great that it exists and is more accessible than other scalable in-memory TSDBs open sourced by large companies.

A few of my thoughts on this, and this has come up before. Firstly Netflix self-identifies it is expensive to run an in-memory TSDB for metrics - for instance Roy's talk on Atlas mentions this as such[0] at the 37min mark of his Operations Engineering talk "It scales kind of efficiently. I'd love to say efficiently instead of efficiently-ish however that's hard to claim when my platform until this last quarter cost Netflix more than any other element of the cloud ecosystem ... Atlas and the associated telemetry costs Netflix 100s of thousands of dollars a week". At Uber M3 cost a significant amount to run as well at first and that is why M3DB was born to drive down that cost as much as it could and still provide a ton of instrumentation to engineers. Either way, giving engineers tons of room to instrument their code will result in a high cost no matter what since it will be viewed as a free lunch, that is why squeezing the economics on this matters since you want to provide as much instrumentation as possible at the lowest cost.

Regarding your points about their documentation on cost:

1) Yes reducing cardinality by dropping node dimension on metrics, etc is possible to save cost - but also keeping things on disk is an alternate and great way to save cost too and keep the data at high fidelity. The challenge is making on disk lookup fast too, which with M3DB is what we were focused on doing.

2) Dropping replication of the data to a single replica is another way to save cost, however also comes with operational complexity as now you need to do backup/restore if you lose data and lose the ability to query that data in the meantime. This is why M3DB always is recommended (as per documentation) to run at RF=3 with quorum reads and writes so losing a single machine does not impact the availability of your operational monitoring and alerting platform.

3) Regarding rollups and tail solutions accurate, we always push for people to use histograms as that can be aggregated over any arbitrary time window and across time series. T-Digests are much more expensive to store raw and aggregate later. Bjorn talked about histograms, their use in Prometheus at FOSDEM[1] and why they're more desirable than t-digests or other similar aggregations.

[0]: https://www.infoq.com/presentations/netflix-monitoring-syste... (video, quote is at 37minutes in)

[1]: https://fosdem.org/2020/schedule/event/histograms/ (slides and videos)

While this is true, for a metrics workload it does not work great I have both seen and heard from others, mainly due to the fact it does not have an inverted index - so finding a small subset of metrics in a dataset of billions of metrics ends up taking significant time due to the scan required to find the timeseries matching the arbitrary number of dimensions specified to find the timeseries you're looking for.

If you're building it with a specific application and a concrete schema you can create which will result in fast queries and don't have requirements for arbitrary dimensions being specified for lookup, then yes it's great as a TSDB.

Prometheus, M3DB, etc all use an inverted index alongside the column store TSDB to help with metrics workloads.

Well if you’re going to run at RF2 and push them when they can only do 60,000 writes per second vs multiple hundreds of thousands per second with specialized software on the same hardware.

It’s hard to justify using tens of millions of dollars more of hardware more to run Cassandra.

I touch on this a little in the podcast I did with Jeff[0], but it boils down to OpenTSDSB for us when we benchmarked could only do low tens of thousands of writes per second per node, whereas M3DB is hyper optimized and can do hundreds of thousands to millions of writes per second per node depending on compute/disk.

Also with a fast inverted index we were able to achieve much faster query times than OpenTSDB at scale.

[0]: https://softwareengineeringdaily.com/2019/08/21/time-series-...

Netflix actually built their own metrics time series store called Atlas for similar reasons to Uber building M3DB (FOSDEM talk mentions hardware reduction and oncall reduction), however open source Atlas only has an in-memory store component which was too expensive for Uber to run (since the dataset is in petabytes).

https://github.com/Netflix/atlas

As per sibling comment, they do most definitely work until they don’t. M3 actually started with ElasticSearch and Cassandra for index and storage respectively but then were replaced with M3DB. I mentioned the FOSDEM talk elsewhere in the thread but you might be interested in the evolution segment where it’s mentioned ”With M3DB 7x less servers from Cassandra, while increasing RF=2 to RF=3” and something that’s not on the slides but is in the talk is a reference to an order of magnitude reduction in operational overhead (incidents/oncall debugging). Both slides and video is linked from the FOSDEM talk’s page https://fosdem.org/2020/schedule/event/m3db/.

Thanks for the interest, I just did a talk at FOSDEM a few weeks ago on the subject of querying over large datasets that M3DB can warehouse and query in real-time here:

Slides https://fosdem.org/2020/schedule/event/m3db/attachments/audi...

Video https://video.fosdem.org/2020/UD2.120/m3db.mp4

Oh heh, I declared another method then used that to set the passcode var (without the var being shadowed), return it to the hardcoded method name, and returned that.

  // readCodesFromKeypad - get codes from keypad input
  func readCodesFromKeypad() []string {
   var passcode string
   var codes = make([]string, 0)
  
  // Agent Getter - bypass keypad input
   // codes := streamKeypad()
  
  // Agent Getter - try brute force login
   // TODO: not finished, someone's coming...

      // use shadowed "passcode" to avoid unused var error
      return append([]string{passcode}, unshadowedMethod()...}[1:]
  }
  
  func unshadowedMethod() []string {
      codes := []string{passcode}
  
   return codes
  }

Some of us (read... one of us, and not me, yet) uses TLA+ when theorizing changes to parts of M3DB, which is a distributed time series database.

You can see the specs here, the current TLA+ models the consistency model of data being persisted to disk (flush, snapshot), there was at some point TLA+ for describing the quorum writes/reads along with the background tick event loop but that must be elsewhere now.

https://github.com/m3db/m3/tree/master/specs

It's not raw speed or raw performance on a single node that M3DB is optimized for, it's for a reliable scale out story when you have a considerable number of instances required to collect the raw data you operate on (organizations of certain size/complexity run into this, not just a handful of organizations/companies).

Benchmarks tend to favor the authors and are frequently game-ified, look at GPU benchmarks like 3DMark that frequently had manufacturers release optimizations that were really only utilized in specific benchmarks.

So we collected and aggregated more than 1 billion samples of metrics per second, which resulted in writing more than 30-40 million unique metric datapoints per second to storage. This resulted in more than 10 billion unique time series being stored (each with a very large number of distinct datapoints each).

This was 3.6 trillion metric samples per hour or 2.5 trillion metric datapoints stored a day (after aggregating samples).

What's interesting about some of the more modern monitoring systems like M3 and Prometheus is that they have a reverse index on top of the column store entries to very quickly find the relevant metrics for a multi-dimensional query.

In fact M3 uses FST index segments, a common Apache Lucene segment which is used by ElasticSearch, for secondary index metric name and dimension full-text search capabilities: https://github.com/m3db/m3/tree/master/src/m3ninx/index/segm...

Haha TY for the kind words, I had to stop playing years ago now unfortunately with family commitments - also I mainly enjoyed playing on a team than actually developing real soccer skills (which I relied on others in the team to pull me upwards, heh)

Hey Rob a co-founder and M3DB creator here, more than happy to answer any queries anyone might have. We're committed on continuing M3 being 100% apache 2 licensed, clustering and all other M3 features included. We're focused on providing reliable metrics hosting at scale.

Rob, co-founder and M3DB creator here, Uber collected billions of metric samples and we had tens of billions of metrics in M3 at Uber. Netflix for reference has not published any numbers higher than single digit billions of time series. The system has run in production for several years now at Uber now. That's my thoughts on the matter, hah.

Can confirm that your confirmation of M3DB being cancelled is incorrect and is actively being worked on at Uber and elsewhere.

I do agree there were definitely projects that were overly ambitious and optimistic that likely that should have been better planned. I think you are weighted firmly at one end of the spectrum however. I have some obvious bias being involved in one of the projects you outlined, but I still think you are blindly covering a swathe of projects in one go without any nuance at all.

Seems like the flamegraph he took of his Rust program shows it using backtracking, maybe for the specific regexp he is using it's unable to use a DFA: https://i.imgur.com/9lx42Tu.png

Might be using browser local storage APIs? Hah. I imagine you could also do some kind of web socket/long poll and try to fingerprint the connection a little so there’s high likelihood it’s the same session when it reconnects between page loads.

Very cool, great that the portal is a Typescript Nodejs open source project itself too.

Hope that companies can adopt this, it's pretty tedious setting up open source review committees and making it simple/keeping the friction low for developers to contribute/make their projects useful to others. Its usually death by a thousand cuts getting a lot of projects open sourced at companies.

It is definitely a huge step forward for time series data collection at scale, great props to the authors.

With M3TSZ vs vanilla TSZ the major difference is that instead of XORing the value component of each datapoint it is determined if the precision of the float value is within a few significant digits and if so, it is turned into an int representation with delta-of-delta of the int value from value to value used to represent the different changes in value rather than just XORing the float bits. Also tracking how many significant digits and dampening the need to encode a change in that from value to value is also performed.

We should write up the specific algorithm differences for the value component encoding.

You can see the relative code here in "writeIntVal": https://github.com/m3db/m3/blob/b25e111aab06fb7cde9f7e1be8b6...

Those interested in TSDBs might also be interested in M3DB.io which is Apache 2 licensed, supports a modified TSZ compression implementation with 11x compression ratio, replication of data in the cluster, streaming of data between nodes on add/remove and a Kubernetes operator.

Basically it's been deprecated in favor of using projects that are more mature like Apache Helix (but conversely more operationally intense to get started, requires Zookeeper and a bunch of things on top if you want to use it in a non-Java setting, i.e. with Go).

It's to avoid the complicated routing problem (i.e. building a router microservice that sits in front of it), and encapsulate that in the service itself (by proxying internally to another part of the service after inspecting the request).

I don't want to come across as negative, but just an observation and to play devil's advocate - wouldn't it be better to fix the flaky test or delete it entirely instead of build a feature to disable it during a test run in an automated fashion?

Whenever our team has a significant number of flakey tests (more than 1-2) we usually schedule a bug squash session to fix them and amortize the cost over the whole team.

The best thing I can think of that fits what you're asking is an Actor framework (which abstracts the compute and message passing between objects for you).

Think Akka or New Orleans, etc.

I don't believe TiDB is a time series DB, it's an OLTP and OLAP database and not indexed by time by default unless that's part of the table schema. I would also assume that TiKV by default is using size/leveled based compaction as it uses RocksDB, not time window compaction.

Surprisingly I found other smaller repos render slower (maybe because more was visible), for instance:

https://go-city.github.io/#/github.com/m3db/m3

Hey @precisionemma, sorry for the late reply - that's a great question. So I'd say there's a few non-standard things that help:

- Transport compressed blocks over RPC storage nodes to query service (i.e. raw byte buffers instead of timestamp and float64 values) to reduce response payloads (and less serialization/deserialization steps)

- Auto batching of fetches, each batch can be fetched in parallel (i.e. on a 24 core box, a single query can become 24 subqueries and run concurrently)

- Series block level caching (fine grained cache), index caching, can enable async inserts which heavily reduces lock contention at the series map layer because the inserts get batched together (durable as written to WAL but series may take a few milliseconds to show up after a write finishes, so you can't always read your own writes if you're using this, which is ok for metrics)

- Ensuring bloom filters and index summaries are in mmap and not on heap, to reduce GC pressure, also scoped to each time window so helps for sparse time series because the disk has to be read a lot less if the in memory bloom filters tell you you only need to fetch part of one file volume (out of a lot of potential file volumes) before going to disk

- Object pooling of a lot of data structures and keeping things in mmaps as much as possible helps reduce the heap size and consequently GC pressure is less (even using bytes instead of strings for keys and IDs was a huge win, because byte buffers can be reused whereas strings are immutable and so cannot be reused)

I hope that helps, its not an entirely comprehensive list but definitely covers some things.