I am pretty sure it hurts you in the short run too. After all, the microcode probably does a register rename or the like with a normal swap, but has to actually burn cycles on using aritmetic for the xor swap.
https://en.wikipedia.org/wiki/XOR_swap_algorithm#Reasons_for...