Why wouldn't you use built in partitioning for this?
Timescale has some neat features for handling time series data, but for just sharding out data for scalability I can't see it offering anything substantial. And in that vein, I feel the marketing is somewhat disingenuous in the performance claims. When comparing to vanilla PostgreSQL it is comparing against a very naive usage pattern. People have been getting the same kinds of performance increases from partitioning out time series data for a loooong time.
(Nothing against Postgres - it just that Postgres needs to design for the general use case, while we can optimize for time-series.)
But TimescaleDB is much more than automatic partitioning. You also get:
* 15x native compression (95%+ storage savings) [0]
* Faster queries for time-series (b/c of our optimizations at the query planning / execution level)
* Automatic data retention policies (for automatically deleting old data) [1]
* Continuous aggregates (real-time materialized views automatically refreshed) [2]
(and lots more)
Happy to answer any more questions!
[0] https://docs.timescale.com/latest/using-timescaledb/compress...
[1] https://docs.timescale.com/latest/using-timescaledb/data-ret...
[2] https://docs.timescale.com/latest/using-timescaledb/continuo...
That is a rather remarkable compression ratio. Do you have a link to technical details? The lightweight compression outlined in the Gorilla TSDB and implemented in Uber's m3db is 11x which I thought was best in class: https://cs.brown.edu/courses/csci2270/papers/gorilla.pdf
Here's a blog post outlining our general approach, which applies type-specific compression based on a column's datatype. (While m3db only supports floats, TimescaleDB supports a whole range of 40+ data types.)
https://blog.timescale.com/blog/building-columnar-compressio...
For example, by default we employ:
- Gorilla for floats (+ some optimizations for faster decoding)
- Delta-of-delta + Simple-8b with run-length encoding compression for timestamps and other integer-like types
- Whole-row dictionary compression for columns with a few repeating values (+ LZ compression on top)
- LZ-based array compression for all other types
And the architecture is extensible to support new/updated compression algorithms for different data types. The above are all lossless algorithms; obviously one could drop precision to achieve higher rates.
We also wrote a post describing these various approaches in a bit more detail for folks interested in learning (although a summary compared to the Facebook paper): https://blog.timescale.com/blog/time-series-compression-algo...
Obviously, your compression rate depends heavily on workload, but the number 15x comes from median performance that users see in the field against their real databases, not just a synthetic or single workload.