Also have you tried enabling precompiled bridging headers? This was added in 3.1 with the goal of speeding up mixed Swift/ObjC builds (by none other than Graydon Hoare), but other than that I don't know much about it.
https://github.com/apple/swift/commit/5b9166ba8098a23e0601aa...