XMPP solves many of the issues with email only because it requires explicit whitelisting of message senders, and because it's still a niche ecosystem compared to email. Most users don't even try to accomplish cross-domain message sending, much less extensive file transfers or server-side storage and archiving of messages, no matter what the server capabilities might be.

As a set of protocols, SMTP and IMAP are both pretty well-suited for the task they set out to accomplish. Furthermore, the delays you mentioned in message delivery are largely due to intermediate spam filters and store-and-forward queues, not inherent issues with email as a transfer mechanism.