Like? Because I've never seen XeLaTeX do something that made me go "wow time to go back to the legacy tex-that-doesn't-understand-unicode->DVI->PDF route".
The biggest issues with XeTeX (compared mostly with LuaTeX, some of them also apply for pdfTeX) in my opinion are (roughly ordered for importance)
1. the missing support for microtypesetting features, especially font expansion
2. Missing support for new OpenType features. I consider especially variable fonts to be important, but I know that there are also many users missing color font features.
3. The font selection system which uses completely different lookup paths for font names and filenames leads to extreme confusion for users, especially since it's system dependent.
4. It shares the problem with older engines that many things like colors, underlines, etc. require special invisible nodes which can influence line-breaking in unexpected ways.
5. (From a programmers perspective) adding advanced PDF features under XeTeX tends to be much more complicated because of the DVI based backend which makes it harder to control the PDF file directly.
That being said, I would take XeTeX over pdfTeX any day, but I don't see much reason to use it if LuaTeX is available as an alternative.