> If you play all the tricks Google's Capacitor does to the data layout
can you please elaborate? does duckdb support querying custom storage formats?
can you please elaborate? does duckdb support querying custom storage formats?
No, this is still parquet being loaded in.
The "trick" is running through sort order combinations which result in maximum packing in a Dremel style data format (like Parquet).
Here's the trick explained in enough detail to understand, but not enough to copy as-is
https://cloud.google.com/blog/products/bigquery/inside-capac...
Search for the section with
"Capacitor builds an approximation model that takes into account all relevant factors and comes up with a reasonable solution."
or a more detailed dive in the paper (by the favourite author of performance engineers, Daniel Lemire)