Worse, using voice recognition allows companies to store your voice samples, that allows them to generate your voice fingerprints, that allows your presence to be detected once you happen to say something out loud in proximity of such devices.
I am afraid that soon it is hard for general public to avoid the situation where when they flush the toiled some unexpected device will recognise it and report to the headquarters.
From Apple employee: "only collects the Siri voice clips in order to improve Siri itself"
Which is basically what Google says.
EDIT: both companies have promised "off-line" processing, whatever that means, but I don't know if any already does that. Note that off-line processing does NOT mean they won't store your voice clips on their servers.