Fantastic!

It would be interesting to combine pipelines into something that branches off into various categories. For e.g., you could split the lines of access-log file into IP addresses and size of requests and fork off separate processing threads: one for obtaining unique set of IP addresses and another for summing up.

It seems like all that is needed for describing a pipeline is:

  (a) a queue for input
  (b) a processing program that's connected to the queue
  (c) (possibly multiple) queues for output
  (d) a topology connecting the processing programs
  (e) a job scheduler.

At the face of it, it looks similar to Apple's Automator and Matt Welsh's PhD thesis on SEDA:

  * http://www.eecs.harvard.edu/~mdw/proj/seda/
  * Paper: http://www.eecs.harvard.edu/~mdw/papers/seda-sosp01.pdf

EDIT: Formatting