I'm sure there are many places where other pointer types would be better, but, like I said, I only tend to see shared pointers, usually typecast to something like FooPtr and used indiscriminatly. It's an uphill battle to even use something like a pointer to const.
I've never seen refcounting overhead show up in callgrind, so I think the choice to uniformly use the most general version is OK.