HNHacker News
TopNewBestAskShowJobs

paperwhite

1 karma · joined March 11, 2024

submissionscomments
paperwhite··on Most storage and query efficient database to store 100B rows of data?
Checked

Some attributes are scalar but others are not

There are array of objects in a lot of places. will need to modify the schema to be completely scalar

Will run a local test & see how this goes. Thanks a lot!

paperwhite··on Most storage and query efficient database to store 100B rows of data?
We have Data heavy products

Typical use case:

Filtering billions of 'docs' or rows across 50 attributes in any number of combinations

A doc holds data on the techstack of a url - For one of the products. Ever evolving schema (hence why we picked Mongo)

--

Number of queries are low but the complexity of the queries are high

We've moved object storage from JSON to Parquet already before uploading to Mongo so that's already 1/4th the object storage

Question if something can match the querying performance of Mongo but has better storage efficiency

paperwhite··on Most storage and query efficient database to store 100B rows of data?
Overhead comes from the BSON format

Even after the wiredtiger auto-compression, the basic overhead doesn't have a workaround