Hacker News Reader

Comment by YawningAngel | Hacker News Reader

You can probably use https://docs.aws.amazon.com/AmazonS3/latest/userguide/replic... to accomplish this

aws cli cp can copy between buckets.

That’s true but data flows through the machine executing aws cli. The parent was asking how to copy data without it flowing through a compute instance.

I’d double check the cli code path for that parameter. The s3 api does have a copy method, which performs the operation within s3 without compute acting as an intermediary. If that’s not the case for the copy parameter, sounds like a bug that needs to be fixed in the cli tooling.

It is directly checking for s3 to s3[1] and indicates that it wants to copy...

I've read over it and I'm reasonably sure that it's going to issue CopyObject, but it would take me actually getting out paper and pen to really track it down.

The AWS CLI and Boto are a case study in overdoing class hierarchies. Not because there's any obvious AbstractSingletonProxyFactoryBean, but rather that there's no instance that stands out as "this is where they went wrong" and nevertheless the end result is a confusing mess of inheritance and objects.

[1]: https://github.com/aws/aws-cli/blob/45b0063b2d0b245b17a57fd9...

Not to mention the insane over engineering of a python 2.7-compatible async task stealing io loop which is slow as hell, and pitifully delivers a maximum of ~150MB/s with 30% cpu core activity. That's why anyone needing to regularly download/upload files from S3 need an additional library (s5cmd, s3pd, etc)

Thanks! I don’t know why I had the understanding that it worked the other way. This is useful to know!

No, when copying objects between buckets (aws s3 cp s3://... s3://... and the corresponding sync command), the AWS CLI uses CopyObject (https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObje..., previously known as S3 PUT Copy), in which the client doesn't handle any object contents. The call stack eventually reaches https://github.com/boto/s3transfer/blob/develop/s3transfer/c... (or its multipart equivalent), where it calls the botocore binding for this API.

Reply on news.ycombinator.com