Poor cache utilization on traditional shared memory arch when passing data (packet) from core to core. I wonder if, while going down this road, they made sure key structures were cache-aligned.

Flow affinity, locking "streams" to cores, is the way to go. Relevant linux kernel support: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux.git...