- for OLTP tasks, something row based like sqlite is great. Small to medium amounts of data mixed reading/writing with transactions
- for OLAP tasks, arrow looks great. Big amounts of data, faster querying (datafusion) and more compact data files with parquet.
Basically prevent the operational database from growing too large, offload older data to arrow/parquet. Did I get this correct?
Additionally there seem to be further benefits like sharing arrow/parquet with other consumers.
Sounds convincing, I just have two very specific questions:
- if I load a ~2GB collection of items into arrow and query it with datafusion, how much slower will this perform in comparison to my current rust code that holds a large Vec in memory and „queries“ via iter/filter?
- if I want to move data from sqlite to a more permanent parquet „Archive“ file, is there a better way than recreating the whole file or write additional files, like, appending?
Really curious, could find no hints online so far to get an idea.