HNHacker News
TopNewBestAskShowJobs

lnkuiper

19 karma · joined August 27, 2021

submissionscomments
lnkuiper··on DuckDB 0.9
Stabilizing storage is high priority for us! You won’t notice anything new while using DuckDB, but we’ve actually rewritten our entire serialization framework in this release. This was a big step towards stable storage
lnkuiper··on DuckDB 0.9
We plan to update our results on ClickBench at some point, as we’ve made some improvements there in this release. I’d expect performance to be similar
lnkuiper··on DuckDB: Querying JSON files as if they were tables
This is very true. DuckDB does not support JSON because it’s a good tabular format, but because JSON is ubiquitous, and there are many use cases where querying JSON dumps for analytics is useful.
lnkuiper··on DuckDB: Querying JSON files as if they were tables
DuckDB will use multiple threads for reading the same file. Each thread will read different parts of the file, but the output will be in the order that the file came in due to DuckDB’s order preserving parallelism.
lnkuiper··on Fastest table sort in the West – Redesigning DuckDB's sort
There are plenty of TopN sorts in TPC-DS, but not many (if any) regular sorts, so no.

There are order-dependent window functions in there though, which did show a speed up.

lnkuiper··on Fastest table sort in the West – Redesigning DuckDB's sort
You are right, we could probably re-run SQLite purely in memory, but only because macos dynamically allocates additional swap.

However, I would not expect much improved performance, because I do not believe that SQLite has a different sorting strategy when running in memory. It would only save some i/o operations, which are very cheap on the macbook anyway.

Either way, would be an interesting experiment.

lnkuiper··on Fastest table sort in the West – Redesigning DuckDB's sort
Exactly. Even when it’s inlined, having a comparator with if/else is much, much slower than a single memcmp
lnkuiper··on Fastest table sort in the West – Redesigning DuckDB's sort
Customer fits in memory, whereas catalog_sales does not.

We chose to remove SQLite from the results because it was so much slower. The plots are much less readable when they are stretched out by something that is slower by an order of magnitude

lnkuiper··on Fastest table sort in the West – Redesigning DuckDB's sort
I'm not sure if I understand, but we did not use a memory map (mmap), but rather blocks of memory that are explicitly (un)loaded by the buffer manager.

The M1 + SSD performs really well here. We tried to an external sort experiment on the x86 machine, but the SSD is old and only has a write speed of 150MB/s (compared to the MacBook's 3GB/s) and it was incredibly slow. So you definitely need a fast SSD for this.

The columns we chose to sort by are rather arbitrary, but we shuffled the table before running the experiments to make sure there is no ordering left from the generation in there.

I like the merge sort trick you described!

lnkuiper··on Fastest table sort in the West – Redesigning DuckDB's sort
We wanted to use the same setup for all experiments, so we had to choose for an on-disk DB for SQLite, because TPC-DS SF100 catalog_sales does not fit in 16GB memory.