DiscreteTom

As per AWS said in https://aws.amazon.com/cn/blogs/aws/aws-lambda-functions-now...

> Each synchronously invoked Lambda function now scales by 1,000 concurrent executions every 10 seconds.

I tried to spread large dataset into thousands of files on S3 and use StepFunctions Distributed Map to launch thousands of Lambda instances to process those files in parallel, using DuckDB (or other libs) in Lambda. The parallel loading and processing is way faster than doing this in a single big EC2 instance.