That's not entirely true, if your decrement results in a cascading free of an entire tree of objects you will pay for the deallocation time for that entire tree -- decrement, free, decrement children, free, etc..
And unless your RC algorithm is more clever than most that's going to be stop-the-world.
Rc is scoped to a single thread, so it'll be, at worst, stop-the-thread.