Hacker News Reader

Comment by brryant | Hacker News Reader

I have no doubt that whenever Salvatore releases a distributed, Redis queue feature it will be amazing. But if you're on AWS, SQS is a solid at-least-once delivery queue service that is hard to beat. (at least when you don't want to manage your own redis machines)

View on HN

What is your fallback though if amazon loses your confidence? AWS infrastructures seem very nice, but aren't you hitching yourself to their wagon in a way that doesnt give you an easy or cheap exit strategy if things go south?

I keep hoping one day that they will open source the software that runs everything so that worst case scenario you arent SOL if you want to migrate away. Some might say that would lose customers for them, but it would probably gain people like me.

Restrict yourself to a set of simple, generic queue API, .e.g. put, getNext, getWithID. Abstract those APIs and implement with them with SQS first. It's fairly easy to switch to a another implementation (Kafka, activemq, etc.) later. Even in production, you can easily switch off the producers, drain the queue, move to the new queue, then restart.

If amazon loses my confidence I think I'd start shopping at Walmart before I switch infrastructure platforms... I see AWS as a century old business, just in the same way PG&E and utility companies of the 20th century have developed a stronghold in energy distribution.

But if I had to host one myself, http://nsq.io/ is the way to go.

If you have a lot of tooling targeting AWS, but want some reassurance for the tools (i.e. not speaking to the data, merely the APIs) then Eucalyptus (https://www.eucalyptus.com/) which is Open Source (https://github.com/eucalyptus) under the GPLv3 might be a way out of that lock-in.

I haven't yet taken the time to provision a local Eucalyptus cluster to test the actual versus the marketing, but it certainly seems like they have a lot of momentum which I wouldn't expect if it didn't do as advertised.

SQS is certainly simple but I've had race condition issues with it in the past.

Are you deleting messages after you receive them? I haven't run into any race conditions but I suppose if I did, I'd use redis's HyperLogLog or something similar to see if I've already processed the message.

HyperLogLog is for cardinality estimation, not set membership. A bloom filter would be more appropriate I believe.

Reply on news.ycombinator.com