The Seven Deadly Sins of Cloud Computing Research (2012)

Sin #1 - Unnecessary distributed parallelism: It does not always make sense to parallelize a computation. Doing so almost inevitably comes with an additional overhead, both in runtime performance and in engineering time. When designing a parallel implementation, its performance should always be compared to an optimized serial implementation in order to understand the overheads involved. If we satisfy ourselves that parallel processing is necessary, it is also worth considering whether distribution over multiple machines is required. The rapid increase in RAM and CPU cores can make local parallelism economical and worthwhile. Establish the need for distributed parallelism on a case by case basis.

Sin #2 - Assuming performance homogeneity: Virtualized cloud environments exhibit highly variable and sometimes unpredictable performance. Base reported results on multiple benchmark runs, and report the variance.

Sin #3 - Picking the low-hanging fruit: It is unsurprising that it is easy to beat a general system by specializing it. Few researchers comment on the composability of their solutions.

Sin #4 - Forcing the abstraction: In some application domains, it is unclear if a MapReduce-like approach can offer any benefits, and indeed, some have argued that it is fruitless as a research direction. Ideally, future research should build on [iterative processing, stream processing and graph processing], rather than on the MapReduce paradigm.

Sin #5 - Unrepresentative workloads: The common assumption in academic research systems is that the cluster workload is relatively homogenous. Most research evaluations measure performance by running a single job on an otherwise idle cluster.

Sin #6 - Assuming perfect elasticity: The cloud paradigm [...] its promise of an unlimited supply of computation. This is, of course, a fallacy. Workloads do not exhibit infinite parallel speedup. The scalability and supply of compute resources are not infinite. There are limits to the scalability of data center communication infrastructure. Another reason for diminishing returns from parallelization is the increasing likelihood of failures and vulnerability to "straggler" tasks.

Sin #7 - Ignoring fault tolerance: Many recent systems neglect to account for the performance implications of fault tolerance, or indeed of faults occuring. For each system, we should ask whether fault tolerance is relevant or required. If it is, it makes sense to check precisely what level is required, and what faults are to be protected against; consider and ideally quantify the cost.

Its good to see something like this out there. There are literally thousands of papers published which try to demonstrate "speedups" but which are usually not very reproducible or useful. The "publish or perish" mentality is responsible for a lot of this. The only way to be upto date with cutting-edge meaningful research seems to be to attend conferences and talk to other researches in the field, and try to publish in the better journals.