Swift 5 Release Notes for Xcode 10.2
developer.apple.com
developer.apple.com
Swift's progress also has little to bear on that front as UIKit and SpriteKit are written in C, Objective-C, and C++; their use in Swift depends on Apple's Objective-C runtime which is not released as part of the non-macOS/iOS port of Swift for Linux.
So faster load times and smaller app bundles!
Each version up to 2 had a lot of language breaking changes, 3 had incredibly rough standard library changes, and everything since then has been about 5 minutes of work.
You can alternatively import Python and use NumPy directly, as well as any other C library, or you can define numeric operations in Swift. (On the other hand, if you are using fixed-size 3x3 or 4x4 matrices, the simd_floatNxN types deliver vastly better performance than NumPy can. So it depends on what your goals are.)
There are plenty of other compelling reasons to prefer Swift over Python such as its type system or its easy, somewhat native, integration with existing Swift, Objective-C, and C frameworks available for macOS, iOS, watchOS, and tvOS.
Speed isn't the only driver, and I'm sure not it's not "the [only] point".