Swift compile times will never be close to ObjC or C compile times, no matter how much engineering goes into it. This is because of how the language is designed, it requires a typechecker which solves an NP hard constraint solving problem.
What swift compiler devs complain most about is about how operator overloading causes some sort of O(n^k) or worse check, because things like the '+' operator has dozens of implementers.
If your really worried about type inference slowing you down, you can write out all of your types like you didn't have type inference. You could even do it in an automated fashion like your source code was some sort of cache.