It seems to happen on heavily overclocked CPUs. Phoronix user had managed to replicate the issue he wasn't experiencing by simply pushing it a bit further.
CPU temperatures were in the upper 50s; downright cold compared to the rather hot (and old) Xeon CPU it replaced.
Interestingly enough, and I'm not the only one to report this, rebuilding the entire system with GCC 6.3 caused the problems to go away (I'm running Gentoo, so this was quite feasible). This is really odd because I was not using any AMD specific cflags, just the default x64 march.
I'm guessing the problem didn't actually go away, but rather the instruction scheduling of GCC 6.3 is less likely to cause whatever the underlying problem was.
segfaults happen even with factory defaults.