Hacker News Reader

I don't understand this comment. You can run arbitrary code in AWS lambda, you can just run "tar". Why did you need to reimplement it?

View on HN

I suppose they are streaming the tarball directly to S3 using multipart uploads, maybe parallelizing this task at the same time.

You wouldn't be able to do it using regular tar.

hit the nail on the head

Yes, you can, you can pipe tar to the AWS CLI to upload a stream, like "tar -c ./myfiles | aws s3 cp - s3://my-bucket/myobject".

For very large uploads you must also specify a chunksize using "multipart_chunksize" such that the upload will require 10,000 chunks or fewer (the CLI can't do this for you because it doesn't know the size of your stream up front).

Wouldn’t you have to have an ec2 open ? We are trying to minimize infrastructure and do it all in lambdas

Nothing prevents calling the AWS CLI inside a Lambda.

It wasn't always like that though. I think they added custom runtimes around 2yrs ago... Your options were very limited before that.

Before custom runtimes it was still possible to ship your own binaries and shell out to them from the official handler function.

Reply on news.ycombinator.com