Ask HN: How do you do batch processing?

I'm curious to to hear what technologies others use to process large volumes of batch jobs.

We've found it useful to be able to offload more process-intensive jobs from an end-user execution, and pass it into a queue ( which looks something like this http://abhinavsingh.com/blog/2010/02/memq-fast-queue-implementation-using-memcached-and-php-only/ )

What do you use?

View on HN

Resque. https://github.com/blog/542-introducing-resque

I use the Ruby version, but there is a php version too.

https://github.com/chrisboulton/php-resque

We're using a combo of the PHP version and the Ruby version and it's been working great for us (we migrated over from a custom queue on top of MySQL. The Sinatra based front-end is awesome for being able to monitor everything that's going on and with a little bit of work you can get pretty much any of the Ruby extensions to work with the PHP version (we're using the resque-scheduler https://github.com/bvandenbos/resque-scheduler to do delayed jobs). There are a few inconsistencies between how the jobs get handled between the two (it's not a huge difference and I've been meaning to submit it as a patch, I need to separate out some of the changes I made to make the PHP version compatible with resque-scheduler).

If you have questions about implementing the PHP version, feel free to ping me.

There is also a Java version that I wrote a few months ago if that's more your cup of tea: https://github.com/gresrun/jesque https://github.com/gresrun/jesque-web

Reply on news.ycombinator.com