It seems to be some sort of not-quite-hardware thing where they download a neural net, based on what I remember hearing about the implementation and that it takes ~10min and flashes a lot of warnings when you switch from "Alexa" to "Amazon" as the available watchwords.
Not that this is completely secure, of course, but it does suggest it'd be non-trivial, at least.
Couldn't the same be said about the phone in your pocket as well?
It could, yes. Perhaps everyone could be a little more suspicious of phones as well.