Hacker News Reader

I'm not surprised, Parquet's columnar encoding and compression won't really kick in significantly for smaller files.

But with pickling you can only read the data in Python.

I just did some benchmarks and it's pretty similar for small files. The difference would only be noticeable if you're serializing a ton of small files.

Huh, makes a pretty big difference for us. We were using pandas' built-in to_parquet though, which seems to suffer from some overhead.

If pickling is what is working best for you, it can't be much data.