Author response to “HPC is dying, and MPI is killing it” objections

MPI is popular in the HPC community, not just in academia. If you are in the Big Data community, you might not ever have used it before...since Big Data is not really HPC. You wouldn't use Hadoop for a HPC job just like you wouldn't use MPI for a big data job! Heck, you might even use CUDA if your job is parallel enough with a small enough memory footprint (and there is interesting work on combining CUDA and MPI).

MapReduce is kind of limiting, and HPC jobs require more pinned machine allocations anyways (if you schedule them like big data jobs, they perform like crap).

It gets an incredible amount of use in the physical science and applied maths communities, and that's not only in academia. It's an extensive standard for message passing that allows distributed (non- shared-memory) computations. Multiple implementations of the standard exist. It is so widely used that entire software suites (Trilinos and PETSc, neither developed by academia but largely by the DOE) are built on top of its message passing framework.

The author's point is that because it is such a large standard it is difficult to grasp and he would rather use something else.

His opinion notwithstanding though, things are unlikely to change significantly, at least in the near future: even just today we saw an announcement of a massive new specialized cluster being built at Argonne. You can be sure most application programming will be done with MPI, not Spark. Commenter jeffsci on the original post is on point.