For s3 and local filesystem, couldn't you also install aws-cli and do a sync b/w the 2 ? I do that for my stuff. Any reason you wrote this and not use aws-cli ?
"I started with "s3cmd sync" but found that with buckets containing many thousands of objects, it was incredibly slow to start and consumed massive amounts of memory. So I designed s3s3mirror to start copying immediately with an intelligently chosen "chunk size" and to operate in a highly-threaded, streaming fashion, so memory requirements are much lower.
Running with 100 threads, I found the gating factor to be how fast I could list items from the source bucket (!?!) Which makes me wonder if there is any way to do this faster. I'm sure there must be, but this is pretty damn fast."
It's a shame the S3 Api doesn't let you order by modified date, or this would be trivial to do efficiently.
the main innovations in s3s3mirror are (1) understanding this & going for massive parallelism to speed things up and (2) where possible, comparing etag/metadata instead of all bytes.
so far, it has scaled pretty well, i know of no faster tool to synchronize buckets with millions of objects.
I haven't looked at your project, but based on what you've said I agree the way you're doing it is conceptually as fast as it can be (massively parallel and leveraging metadata) whilst being a general purpose tool that "just works" and has no external dependencies or constraints.