Hacker News Reader

Comment by openasocket | Hacker News Reader

This is a neat way to deal with it, and has numerous other benefits, but I thought I'd add another thing to try: deploying your data in multiple regions. You can set up a secondary bucket in a secondary region, and configure your primary bucket to replicate data to the secondary bucket automatically. And then set up your infrastructure to switch to the secondary bucket for read operations should there be a problem with the primary. With large amounts of data the costs could add up, but it has the advantage that you can still serve all read requests during a regional outage, not just the reads that are in the cache.

Obviously this method has cost associated with it, so you should probably only do this if you need complete data availability.

View on HN

Or we can do what we did and save money instead and gain the guarantees. Plus achieve our original goal of performance by bringing the bytes inside our datacenter instead of going out through public internet.

Yeah, what I mentioned does nothing to improve performance, just thought I'd mention it as something to think about. For your use case S3 replication doesn't really give you extra availability. In practice I would likely use your caching method, and use S3 replication in addition, if I had a large dataset that doesn't cache well. Or if I absolutely needed to maintain write availability, in which case I would use bi-directional S3 replication.

Definitely. If our dataset was absolutely massive and we couldn't hold a reasonable amount on disk, it'd make more sense. Fortunately, we are getting a 90+% hit ratio out of a very small amount of space relative to the size of our bucket.

And yeah, we have 0 resiliance for write data here. Again, fortunately, we can afford this tradeoff since the amount of uploads is significantly lower and much less critical for us.

> configure your primary bucket to replicate data to the secondary bucket automatically

Well, when the primary goes down, your write operations would get busted.

If the upstream source allows it, just write to both buckets at once. E.g. with Logstash this is trivial.

I put this elsewhere in the thread, but you can also set up bi-directional S3 replication. I haven't used it in production, but in theory it would mean you can continue write operations during failure. And those writes that are committed to the primary but not replicated when the region goes down wouldn't be lost, they'd come back up once recovery is complete (S3's SLA for data integrity is a lot stricter than it's uptime). Whether or not that is acceptable depends a lot on your use case.

Sure, though in many of S3's use-cases you won't care about writes at all, just read availability. E.g. for S3 buckets serving as canonical binary-asset hosts for CDNs to front.

Reply on news.ycombinator.com