I think Intel is probably planning to unify CPU and GPU. Intel has been bringing GPU-like features to CPUs like very wide SIMD, gather (vectorized memory load), etc.
Just scatter (vectorized store), a 2D-optimized cache controller (for trilinear filtering and hardware swizzling) and fast HBM memory are missing from the latest Skylake CPUs. Otherwise current crop of CPUs can pretty much be a GPU. An 18 core 2.3 GHz Haswell Xeon has 1.3 TFLOPS peak.
> CPUs are WAY faster than most people realize but the way current software is written leaves anywhere from 10x-100x the performance on the table.
Yeah, just a single Haswell core can do 32 (FMA) flops per clock cycle. Other then specialty optimized software, it might see a flop or two every few cycles while being idle a lot.
On the other hand, most software can't be 10-100x faster. 2-10x is pretty common, though. Compilers are still somewhat bad at using SIMD cleverly. Sometimes they do surprise.
> Right now the best option is ISPC, Intel's compiler.
Right now the best option is to still to use assembler or intrinsics (when compiler does ok job at instruction scheduling), at least if you need serial performance in "normal" applications. ISPC is of course nice for very parallel problems.