It also doesn't help that these APIs are rapidly changing, in ways ranging from minor rearrangements that the XCode auto-upgrader can mostly handle, to real porting hassles. I've been working on a project that was started in early 2016, so it was originally in Swift 2, the then-current version. A ton of things changed for Swift 3, which came out 6 months later, in late 2016. And a bunch of things are changing again with Swift 4 (although not as many as in the 2->3 upgrade). I haven't yet looked into Swift 5, but presumably that brings yet more API changes.
A lot of time is spent just on this treadmill of porting previously working things to modified APIs. To make it worse, you have to manage a 4-way compatibility matrix between Swift versions, XCode versions, iOS versions, and macOS versions, where only very specific combinations work.