If your encryption is strong and end-to-end, it doesn't matter where your data goes.
For example, if you use "voice assistant" from some major company, what prevents it from voluntarily sharing all the records with the government for the sake of national security? What prevents its employee from secretly sharing the data under some legal obligations?
The location of the server is really important here.