how can the multicore AVX implementation do a dot product (for arrays much larger than cache) at 340 GB/s on a system with RAM bandwidth < 50 GB/s
The author has updated the post with corrected AVX measurements, with the original ~340 GB/s revised down to 31.7 GB/s. (Thanks CowFreedom)
But about bandwidth, matrix multiplications happen mostly in cache and that has a lot more bandwidth than RAM. Blocks of the matrix are loaded to cache (explicitly in CUDA) and used multiple times there.
I'd exploit the better multi-level cache hierarchy in CPUs and make the code NUMA aware. But still I wouldn't bet against a recent GPU card.
The post is about dot product, not matrix multiply. Dot product has no data reuse