Wouldn't that be pretty slow? :) Why not store it on the EC2 instance directly? Or locally on your computer?
1 TB is .. not a lot of data .. I rent a server with 15 TB for ~50$ per month and buying a new 2TB disk is less than 100$.
1 TB is .. not a lot of data .. I rent a server with 15 TB for ~50$ per month and buying a new 2TB disk is less than 100$.
Disclaimer: I work at DuckDB Labs.
[1] https://duckdb.org/docs/data/parquet/overview#partial-readin...
Since the data in S3 receives updates every few hours, querying it directly ensures queries are on up-to-date data, whereas creating a copy of the dataset on the EC2 would necessitate periodically checking for updates and moving the deltas across to the EC2 (not insurmountable; but complexity worth avoiding if possible).