Moving 25TB data from one S3 bucket to another took 7 engineers and 2 full days

RClone. Probably the best tool I have come across for interacting with cloud storage solutions.

Move Docs: https://rclone.org/commands/rclone_move/

S3 Docs: https://rclone.org/s3/#amazon-s3

Supports parallel server-side copies and deletes (no server-side moves, unfortunately) so this would have been much faster.

Simply copying lots of files over an extended period can break your applications elsewhere:

"Amazon S3 automatically scales in response to sustained request rates above these guidelines, or sustained request rates concurrent with LIST requests. While Amazon S3 is internally optimizing for the new request rate, you might receive HTTP 503 request responses temporarily until the optimization is complete. This might occur with increases in request per second rates, or when you first enable S3 RTC. During these periods, your replication latency might increase. The S3 RTC service level agreement (SLA) doesn’t apply to time periods when Amazon S3 performance guidelines on requests per second are exceeded."

That's a fun gotcha...

I used to work for a company which used S3 as its data backend - copying lots of files in and out of S3 is how the product worked. It was also how we enabled QA/STG and the CI/CD pipelines to interact with some of that data - sanitizing it was just a matter of removing or rewriting metadata.

We never solved this problem well.

Our first go was a naive `aws s3 cp --recursive` between the source and destination buckets - this executes all files to copy sequentially. Not good for performance when you have hundreds of thousands of files.

Our next go was to use some custom Ruby scripts to do it, executed under JRuby so that we could take advantage of Java threads to multithread the process. This also didn't work well - the Ruby SDK's `CopyObject` implementation seems to hang some threads if you transfer multiple files concurrently. Our team was primarily Ruby-based, so going pure Java was an option that we didn't pursue.

We then heard about S3 Batch Operations, and launched an unsuccessful POC, because Batch Operations don't work on files larger than 5GB. That's an immediate non-starter for our use case.

We ended up with a horrible hack after many person-weeks of work on this problem: 1) Get a list of all of the files in the source bucket, and put them on a queue. 2a) If a file is small, copy it using a native Java/Ruby threaded worker. 2b) If the files is large, fork an `aws` CLI process and have it `s3 cp ...` the file.

I'm not surprised at all that copying data took these seven engineers two full days.

I’m surprised none of the engineers either knew about, or thought to ask AWS support, S3 Batch Operations or S3 Replication.

Both of which could’ve completed this process in a relatively short amount of time in a completely hands-off way.

When I Google for “move large S3 bucket” the first result I get is an AWS article which goes through all the best options: https://aws.amazon.com/premiumsupport/knowledge-center/s3-la...

GCP and Azure both have much better built-in tooling that would make this a few clicks. Their storage system design is also much better. It's unfortunate that the industry has standardized around S3 just because it's a first mover rather than pushing Amazon's product to get better.

Fun problem. Without replication, here's how I would do it.

A pipeline: S3 list worker(s) (see below for optiizations) -> SQS -> aws lambda (s3 sync) -> SQS -> aws lambda (s3 delete of original key).

Now the secret: to make it really go vrrooom you need to sufficiently parallelize the list operations. For this enable S3 storage analytics on the bucket to get a manifest of all of the files a few days beforehand. Use that list to partition the prefix/key space evenly into 20 or 30 (or 100, or 200) workers each with start and end key prefix.

With SQS, you can make sure the sync and delete actions succeed, and with lambda they'll run in a serverless fashion as fast as you can feed the queues.

Amazon will absolutely throttle you when making a lot of requests. Reach out to them ahead of time. Make your lambda code robust, so that a failure is idempotent lean into the retries.

Something sounds fishy about this. What exactly were the 7 engineers doing during this time? And none of them thought to Google it or call support?

TBH, I won't claim I know the best way to do this, but I'm 100% sure a call to Amazon support would sort things out. Copying data between buckets is common enough I'm sure they have a good way to handle this.

Even if you were moving 25TB of data on a server under your physical control, it could be a significant engineering effort. Maybe not 14 person-days but at least a few.

What's amazing is that some teams assume the same operation will be more performant in the cloud, despite being wrapped in layers of HTTP-based API abstractions.

TL;DR - The delete is the hard part. You can use S3 batch operations with an inventory list to do the copy very quickly. Alternatively, you can setup replication and "touch" each file using a self-copy CLI command, once the replication policy is in place. For the deletion, you're sort of stuck with lifecycle policies, which would take a day or so to clear out the old bucket, but could be supplemented with manual interaction, I'd imagine. There probably needs to be a better mechanic for completely wiping buckets. Last I checked there was not.

I had a similar task once, tried it with awscli and s3cmd first, then took about 5 minutes to google for a multithreaded alternative and found s4cmd[1], problem solved. (tmux to keep the session running)

Makes me kinda doubt at least parts of this story - not one of these 7 engineers thought about googling?

Edit to add: Agree with the basic premise though - s3 can be real slow when when you actually need to work with your data.

[1] https://github.com/bloomreach/s4cmd

Lots of weird comments here. I’ve done this solo with a 200tb bucket in about a day. A rando “aws s3 cp” obviously isn’t going to cut it.

They had the weird requirement where they also needed to empty the bucket, which makes things slightly more complex. You’d create a small 5 line lambda that would copy an object to the new bucket and then delete it. You’d then invoke this with a batch operation.

If you can relax the empty requirement, then you’d just use the batch operation to copy without needing a lambda and set a lifecycle policy to delete the objects. You’d need to do a bit of manual work to copy-delete objects created since the last inventory ran, but that is fairly simple.

You can use a tool like rclone but that’s still going to be quite slow compared to a batch operation, especially if you have a higher number of small files.

Alternatively, you would set up replication between the buckets and just handle the deletion within the window. S3 can delete 1,000 objects per call, meaning emptying a bucket is pretty fast.

Or, lastly, you’d use “s3 cp” with an appropriate number of threads on a machine within AWS.

I had a similar problem, except I had to download about 600GB (mainly from small <30kb files) from a client's s3 to on-prem. It took a painfully long-time using the AWS-CLI.

Never really found any good solutions but fortunately it wasn't urgent so it wasn't a big deal. But this should be easier.

I often copy (not synchronize) remote files using rsync instead of scp because rsync will batch the small files, and scp does not.

Or zip up (or tar) the small files, copy over the large archive, then unzip.

They are having the same issue here: A lot of small files each transferred one at a time, instead of in a streaming batch.

Do you hear the sound of clinking chains? That’s vendor lock in ;-)

That's because they did it in the worst possible way.

AWS has built in tools to do this in the background where they take care of everything for you.

I moved a ton of data in and out of S3 when I worked at $STARTUP. You just need to tune the s3 options a bit. Haven't had any problems using a beefy machine, often saturating 1Gbps with small files, or 10Gbps with large files, using a single beefy machine.

We are all hostages of the clouds!

Yikes. I miss Unix and rsync

I think you can use replication to accomplish this.

They have Snowball for this, up to 50 TB. https://aws.amazon.com/getting-started/projects/migrate-peta...

Also, throwing a swarm of lambdas to chunk the files into about 250mb temporary zip files would help.