Hacker News Reader

Comment by ummonk | Hacker News Reader

Yeah S3 has similar performance issues where accessing objects with the same prefixes has lower throughput because they get sharded onto the same server. It's very counterintuitive when you're used to how performance works on single computers where you want to optimize for cache-locality.

View on HN

This used to be the case, but it's not true any more.

For example, previously Amazon S3 performance guidelines recommended randomizing prefix naming with hashed characters to optimize performance for frequent data retrievals. You no longer have to randomize prefix naming for performance, and can use sequential date-based naming for your prefixes.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimi...

The doc is talking about how the full prefix is now part of the sharding, in contrast to previous times when only the few characters mattered.

But if you put many files with the exact same prefix - even sequential dates - then you hit the threshold the doc says, about 5000 ops/sec.

I re-read your comment and you do say "same" prefixes (I think I read it as "shared"), which AWS hasn't changed the behavior of (AFAIK). You're right that objects with the exact same prefix are still routed to the same shard (set?) and have that throughput limit.

P.S.: In a few projects I've built prefixes using timestamps, but not at the very beginning, and worried that they weren't getting sharded out. The change I linked to fixes that problem.

Ugh really? Why would they not hash the whole filename for shard assignment?

In these systems you typically can list blobs based on prefix. If you shard blobs sharing prefix to different nodes that becomes difficult.

Because the file name includes the "directory" prefix, i.e. each file's name stores the `/entire/bucket/dir/tree`, which can get large.

The size of the input doesn't affect the hashing.

It affects the runtime of the hashing.

Reply on news.ycombinator.com