There is a new open source columnar lakehouse gaining traction with low cost data processing and storage, with Polars as part of it:

1. Querying and Data processing in Polars or DuckDB

2. Metadata (for transactions, time-travel) table formats (Iceberg, Hudi, Delta)

3. Storage with Parquet on S3.

With Polars/DuckDB, you can process up to 100+ GB of data at a time on a single VM, turn it off when you're not using it. It's a lower overhead, lower time-to-value stack than Spark. And even if you have TBs of data, so long as you only process ~100GB at a time, it works fine.