While some may prefer a knowledge about you or you via people like you, as in, "Please recommend me a great movie", Siri is currently not in a stage where she may tell you stuff like "How do I mix a White Russian?" Google Assistant give me 6 steps with a photo and a follow up question "What about Screwdriver?" Siri gets utterly wrecked in the knowledge based questions.
I think the main problem is not privacy-related, but knowledge base-related. Google is building upon a fricking huge search engine via a knowledge graph and the sky is the limit for how well an AI can do. Apple is building on what, a shut down Ping social network, Apple Music listening habits, Wolfram Alpha hopes if all else fails, and a sparse Bing Search API if that failed too?
Not to mention that Apple positioned itself as a brand for people who "Think Different." I don't think people who got taken in by that messaging would be attracted to the prospect of services that can more efficiently pigeonhole them.
I periodically delete all my history from google servers, thanks to their privacy tool, and google gets just a dumb for me as siri.
If you watched the HomePod reveal you'd know that it sends your speech to the cloud for understanding. Which seems like a pretty clear admission that this can't all be done on-device easily.
I believe the restrictions imposed by keyword-spotting have more to do with the always-listening and/or power-efficient nature of the task, rather than the restrictions on the legibility of offline speech recognition.