It is annoying, but they had far bigger problems to fix from release until now. When Swift was released SourceKit would crash all the time. The compiler would crash when code wasn't correct. The error messages were terrible. The language had was missing essential parts. Apple had to do a lot of fixing to get to where we are now. The things they have fixed has been a lot more important to get right than the refactoring support. Crash free IDE, good error messages, stable compiler, user friendly handling of optional types has been far more important IMHO. But now the language is looking good, so I think starting with Swift 3.0 I think we will start to see xCode getting fixed.
It might also be intentional to give a marked to AppCode and others. xCode has always been pretty slim in the offerings when it comes to refactoring. But xCode contains a lot more tools than what you get in e.g. AppCode. There is considerably more effort going into creating good GUI designers, UI Automation tools, performance diagnostic tools, debugger, playground etc.
AppCode has a much smaller scope and can thus focus more on being a great tool for refactoring.