From a PL / compiler standpoint it's pretty obvious that Swift is not going to match Objective C from a performance standpoint. That's why Apple used a micro benchmark comparing a static, compiled language vs a dynamic, interpreted language (Python) for marketing purposes.
> We can’t know exactly what’s going on behind the scenes, but my hunch is some of what we take for granted in Objective-C – the straight C scalar data types – are actually classes in Swift. And the more you rely on classes, the more Automatic Reference Counting is in there somewhere, retaining and releasing like there’s no tomorrow, often for no good reason.
If they switched from primitive types to classes for built-in types, that means vtables and an additional memory lookup per element access weakening caching and memory locality.
Creating a Swift int would involve instantiating a class, incrementing the ARC count, and storing data compared to only storing data for primitive types. That explains why appending Swift's Ints to an array is so much slower than NSNumber.
On a side note, this isn't as much of a problem with Java—despite everything being implemented with vtables—because of JIT and better CPU branch prediction.
Disclaimer: I have no Objective C / Swift experience.