There are quite a few reasons. First of all the recordings made by voice assistants aren't connected to your identity (or at least most people don't expect it to). Uploading it yourself for the world to hear links it to you in the minds of most people.

Second of all most people expect that the recordings made by voice assistants aren't going to be shared beyond the walls of Amazon/Google. A few random contractors you won't ever meet hearing you feels different than the entire world including people you know hearing it.

Lastly, Amazon/Google has a good reason to listen to the recordings: improve the technology so it works better for everyone including you. Random strangers have no good reason to listen to recordings of you, only bad reasons.

It really bugs how most people here seem to go "oh if you like using voice assistants why don't you just share everything with the whole world?" Privacy is a spectrum, not a binary choice. Somebody can be comfortable with Amazon/Google using the audio and not be comfortable with a live audio stream accessable to everyone on the internet.