I had some spare time and tinkered with duckdb with a 70GB dataset, but just getting the 70GB on to the EC2 took hours. Would be pretty rocking if duckdb team could somehow set up a ~1TB sized demo that anyone can setup and try for themselves in, say, under an hour.
Reading the data off s3 will mean you will be slower than offerings like snowflake. Snowflake has optimized the crap out of doing analytics in s3, so you can’t beat it with something as simple as duckdb.
Importantly you need the data in some distributed format like parquet or split csv. Otherwise duckdb can’t read it in parallel.
S3 is high latency unless you use for S3 Express Zones (the low latency version).
We used EFS (not EBS) and it was much faster.
On the setup side, I agree that local (instance-attached) disks should be preferred but does EBS incur an IO fee? It incurs a significant latency for sure but it doesn't have a per-operation pricing:
> I/O is included in the price of the volumes, so you pay only for each GB of storage you provision.
> Each synchronously invoked Lambda function now scales by 1,000 concurrent executions every 10 seconds.
I feel like you'd get away with the whole thing for around $500/mo depending on how much compute was needed?
Most of my work is with clients that don't have any set infrastructure yet, so was curious if anyone had any anecdotes.