HNHacker News
TopNewBestAskShowJobs

th24o3j4324234

1 karma · joined February 25, 2025

submissionscomments
th24o3j4324234··on The best way to use text embeddings portably is with Parquet and Polars
The trouble with Parquet (and columnar storage) in ML is,

1. You don't really care too-much about accessing subsets of columns

2. You can't easily append stuff to closed Parquet files.

3. Batched-row access is presumably slower due to lower cache-hits.

It's okay for map-reduce style stuff where this doesn't matter, but in ML these limitations are an annoyance.

HDF5 (or Zarr, less portably) solves some/many of these issues but it's not quite a settled affair.