Swift is based on the ObjC runtime, which means you have a shared heal and the possibility of multiple threads adjusting counts at the same time.
I'm managing something like this in Go. There are no refcounts, but everything is very much mutable. I'm basically arranging for a span of time where I know nothing unprotected by a channel is going to be mutated, then I simply let every thread in the app that cares to read data from every part of the heap, but only during this span of time. The same technique could be applied to a ref counted app. (It would probably work best for games that have a tick.)
I still think it would be hard to apply to a ref counter app, since you'd need to keep track of change in ref count for later cleanup (thread-local per object maybe? sounds inefficient), but I now will admit that it sounds possible.