The Swift compiler has some serious issues with the way it does type inference. Try compiling code with a line of basic arithmetic on a few numeric literals.
As usual, compile times depend on what you're used to. It is something we're working on improving though; we want it to be very fast! We expect incremental recompilation to move out of nightly soonish; that will help quite a bit.