I'm surprised the version with pointers was only about 2x slower, not 10x slower as some people bashing Java often suggest. And this despite the fact that shared_ptrs in C++ are generally heavier than references in Java/C#.
And even the shuffling doesn't really simulate a realistic distribution in memory, it'd be better to also allocate random memory block in between the objects.
1) Layer of indirection
2) Non-continuous memory
3) shared_ptr overhead itself as you mentioned.
All that and only 2x slower? For many applications performance of this particular operation is a rounding error of a rounding error.
This tells me not to worry about optimizing my iterations across datasets prematurely.