Some of the performance gain was coming from the use of Unsafe in earlier versions of Spark (e.g. Spark 1.5). However, the massive gain you are seeing in Spark 2.0 are not coming from Unsafe. It is coming from this idea we call "whole-stage code generation", which eliminates virtual function calls and puts intermediate data in CPU registers as much as possible (versus L1/L2/L3 cache or memory).

We will be writing a deep dive blog post about this in the next week or two to talk more about this idea.