The author had no idea what they were doing with kdb.
They even admit that they couldn't be bothered to modify their ingestion scripts to not partition their data.
They even admit that they couldn't be bothered to modify their ingestion scripts to not partition their data.
> One note about the results for kdb+: The ingestion scripts I used for kdb+ partition/index the data on the year and passenger_count columns. This may give it a somewhat unfair advantage over ClickHouse and LocustDB on all queries that group or filter on these columns (queries 2, 3, 4, 5 and 7). I was going to figure out how to remove that partitioning and report those results as well, but didn’t manage before my self-imposed deadline.
>The pickup_ntaname column is stored as varchars by ClickHouse and kdb+, and as dictionary encoded single byte values by LocustDB.
But it would be trivial to convert to enum/sym type in kdb+. It's silly to query and group by strings.
But even then, hardware/networking performance and tuning can throw a wrench in the most seasoned/knowledgeable approaches. Users can further bring otherwise solid setups to a grinding halt with unanticipated use-cases.
The only hope when you hit these inevitable road-blocks is that you're working for someone that appreciates the difficulty of the problem.
Looking over the queries, he made some... interesting changes that have him benchmarking oranges to apples.
So yeah, maybe the guy has no idea what to do with kdb - but ultimately having a fast, free and open-source database & query language beats a fast and expensive piece of commercial software.