This seems like the wrong way to look at it?
Focusing on the spots in your code where you’re using RC and concluding you ought to be using GC misses the fact that all of the spots in your app where you’re getting along fine without GC would suddenly be forced to bear the costs of GC.
And RC isn’t a poor man’s GC; Apple concluded that Automated Reference Counting was a significant memory and CPU win over running a GC. Unless you’re littering RCs literally everywhere, you’re probably not doing anything much worse than ARC does in Swift (in fact you’re probably doing better if you don’t do it anywhere the compiler didn’t force it, because Swift has less lifetime info and is forced to be more defensive) – you’re more than likely coming out ahead versus what your runtime costs of a GC would be.
(But if you’re not concerned about runtime memory or CPU cycle costs at all, then sure, Rust is probably at the wrong level of abstraction for your problem)