Always-On Time-Series Database: Keeping Up Where There's No Way to Catch Up

From the article:

> TS It's certainly the case that a lot of CRDTs are really complicated, especially in those instances where they represent some sort of complex interrelated state. A classic example would be the CRDTs used for document editing, string interjection, and that sort of thing. But the vast majority of machine-generated data is of the write-once, delete-never, update-never, append-only variety. That's the type of data yielded by the idempotent transactions that occur when a device measures what something looked like at one particular point in time. It's this element of idempotency in machine-generated data that really lends itself to the use of simplistic CRDTs.

So for this particular use case, the CRDT is simple and they want to favor availability over consistency. I'm not sure if they provide further guarantees, e.g. from the description in the article it seems that temporary holes in the time series would be allowed if updates would arrive out-of-order.

The article also mentions Cassandra, it seems they went for a different design so progress is more quickly visible in the DB (among other reasons I suppose).

> What we wanted was a topology that looked similar to consistent hashing databases like DynamoDB or Riak or Cassandra, but we also wanted to make some minor adjustments, and we wanted all of the data types to be CRDTs [conflict-free replicated data types]. We ended up building a CRDT-exclusive database. That radically changes what is possible, specifically around how you make progress writing to the database.

CRDTs work well for the specific case of machine-generated time-series because your records are samples. There are no complex relationships between records even for the same source. Gaps in the time series are like missing pixels in a picture, there is no specific pixel that is critical as long as you have enough of them to figure out what you are looking at in the moment. This also implies that there are no semantically meaningful update operations on specific records beyond converging values.

Platforms like Cassandra can make some of the design choices they do because they don't support high write throughput (relatively). With sensor data models throughput and efficient use of bandwidth is everything, so minimizing the chattiness of ensuring consistency is critical.

I can't speak for Theo's number (it does seem high) but in my own experience working on similar data models, the typical machine-generated data model size roughly doubles every 6 months on average and has been growing at this rate for two decades.

This implies that average data models today are a million times larger than data models a decade ago. Exabyte scale working data models have been something I've needed to consider in designs for at least a few years. We are surprisingly close to overflowing 64-bit integers in real systems.