It does matter. Because there are many people who have access to the servers and they can cooperate with government agencies voluntarily or involuntarily.

For example, if you use "voice assistant" from some major company, what prevents it from voluntarily sharing all the records with the government for the sake of national security? What prevents its employee from secretly sharing the data under some legal obligations?

The location of the server is really important here.