Pros and Cons of Redis-Resque and SQS

I looked pretty hard at systems to do parallel batch jobs.

One thing I looked at was to store intermediate results in Amazon S3 and control the execution of job steps with SQS.

Although the system couldn't do everything Hadoop does, it worked for a wide range of problems and had the desirable quality that it could transparently parallelize multiple processing phases when it was possible.

I went with Hadoop, however, because I can run a Hadoop cluster in the office or in the cloud, and that's a big advantage.