1. You're right, it can be slow! But it's usually still consistent and that's useful.
2. Hmm, cascading deletions. Is that really a big problem in practice? I'm skeptical because it seems like that would affect C and C++ programs too, but you rarely hear anyone mention it. Maybe Swift tends to use more objects whereas C++ programmers tend to be better at packing stuff together?
3. Fragmentation -- that's true, but again, it affects C and C++ too. I guess for long-running C/C++ programs you're likely to manage memory pools directly. I don't know if that's possible in Swift.
4. Cycles -- weak references work fine for this. I have never had trouble with cyclic garbage in Objective-C. (I mean, I've had leaks, but they're always easy to spot with a leak detector and easy to fix with weak references.)
Overall, it seems to me that reference-counting adds a small but consistent performance penalty, and otherwise should have comparable runtime behavior to malloc/free in C, which is known to work pretty well when used correctly.
Note that Apple got smooth and reliable 60fps performance on the original iPhone, which was extremely resource-constrained by modern standards, using Objective-C, which isn't usually considered a fast language!
On the GC side, it seems like you typically get bursty, unpredictable performance, in both time and memory. Modern GCs work very hard to keep collection pauses as short as possible, but almost inevitably that means keeping garbage around for longer, which means using a lot of memory.