Disclaimer: I work on Google Genomics.
Disclaimer: I work on Google Genomics.
Disclaimer: I work in a neurobio lab but I wish I worked for Google Genomics.
Another features that comes to mind for enabling this sort of work is GCE's support for attaching a persistent disk volume to multiple VMs (read-write for one, read-only for others) -- have each pipeline stage consume from a read-only PD containing the output of the previous stage and write its results to a PD shared with the VM(s) that handle the next stage. Have the VMs communicate with each other for control plane operations ("hey, this directory contains the complete results of stage <x>, one of you stage <y> workers can pick it up" or "I have completed processing the stage <x> outputs for job <5>; they can be removed or archived to GCS."). This strategy also allows independent scaling of the workers responsible for each stage based on the computational requirements of that stage, since a given PD can be shared read-only with many VMs (with limited performance impact in the many-reader case, especially if the readers are not hammering the same parts of the volume).
Another option for truly pipeline oriented processing (assuming the underlying workers can generate data suitable for it) is to leverage something like Cloud Dataflow -- then they'll handle the work balancing for you.
(note: I work on GCE)
Clound Dataflow on the other hand is difficult to use in practice - one of the main reasons is the large footprint of intermediate files (sometimes 10x the size of the input)
Currently we are running monolithic pipelines on the cloud where one instance does the job from start to finish (if it get's preempted - I restart). I plan the next version to be in some way "split-apart" but have not found a good strategy/tooling - any advice?