Passing pointers is not significantly faster than passing offsets into a shared memory pool.
Passing a pointer between threads requires going through the memory system and letting cache coherence algorithms sort out the data sharing between cores (with or without a futex lock/unlock depending on implementation). Passing a SHM offset between processes requires going through the memory system and letting cache coherence algorithms sort out the data sharing between cores (with or without a context switch to the kernel depending on implementation).
It's not that different.