If you turn Siri off, it doesn't activate by accident, and the iPhone still works as intended. If you turn off Alexa, the home assistant doesn't activate by accident because it doesn't work at all.
You can turn it on each time you need too use it, maybe with a trusted voice activated power switch that isn't connected to the cloud.
At that point why bother getting it in the first place? The whole appeal of it is that you can just say "hey google" from anywhere in the house and it would answer you. If you have to walk over and turn it on, you might as well use your phone.