I use both. Conversations works well out-of-the-box on Android. I use Vector on my desktop, Matrix Console Android, and Vector Android, but none of them work very smoothly. I'm hopeful they will improve as time goes on as I tend to use XMPP more since I have more personal contacts that I can communicate with via that protocol, but I also like and use Matrix.
You don't seem to be aware, but there was an XMPP<->IRC bridge long before Matrix existed. It was about as satisfying as the Matrix version, though that's still improving.
I'm running my own matrix server (synapse), and my own XMPP server (ejabberd), and honestly ejabberd was easier to get off the ground. Again, I'm hopeful that synapse will improve with time, but I really think there's room for both to exist since they do have slightly different use cases.