The question, why many scientific communities haven't been racing to adopt Big Data technology is a tough one. Whether or not utilization is a major factor I can't really say. For Cuneiform, we have assumed that the major obstacle for using platforms like Hadoop or Spark directly is that these systems require a library to have a Java or Scala interface. If your software is written in, say, Python you need to either

- write a wrapper (an approach taken by Crossbow [1], SeqPig [2], or BioPig [3])

- reimplement (an approach taken by CloudBurst [4] or ADAM [5])

Reimplementing a whole pipeline represents a huge upfront cost. Writing wrappers is more feasible and, accordingly, a huge amount of custom tailored pipelines and ad-hoc worklfow systems have emerged recently. So if it boils down to wrapping libraries and running them on Hadoop, then we need to make this as easy as somehow possible. Cuneiform is a step in this direction.

[1] http://bowtie-bio.sourceforge.net/crossbow/index.shtml [2] http://bioinformatics.oxfordjournals.org/content/30/1/119.lo... [3] http://bioinformatics.oxfordjournals.org/content/early/2013/... [4] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2682523/ [5] http://bdgenomics.org/projects/adam/