Since it is missing mutable arrays and strings, your code often ends up copying a lot of memory around if you're not extremely careful, and you quickly get accidentally quadratic complexity.
A lot of APIs are just wrappers around ObjC APIs, with lots of memory management calls required for bridging. You really need to be careful what calls you put in a tight loop.
And that's before you even start with the good stuff. Protocols with associated types are so slow that they are unusable for anything performance sensitive. Theoretically the compiler should be able to optimize it incredibly well since it's all statically typed, but in practice the compiler seems to never be smart enough. Call stacks get very deep very quickly (protocol witnesses, closures, ...).
In my experience the easiest way to make Swift code fast is to rewrite hotspots in ObjC.