For unique_ptr, in a release build, no: but in a debug build there's overhead, and more to the point, stepping into the deference in gdb/lldb at least in my experience requires stepping through the internals, to the point some of the code we use with them is #ifdeffed hackily to use raw ptrs in some cases to avoid this annoyance.

For shared_ptr, the atomic ref counting can cause surprising overhead due to contention in multi-threaded scenarios, even if just accessing the pointer, depending on how the shared_ptr is passed through functions...