I suspect your misunderstanding comes from your mistaken belief that these devices provide very little value for their users.
disable smart assistants on your phone and the phone won't start recording you "accidentally". You don't have that option with a smart speaker.
https://thenextweb.com/google/2019/07/10/google-contractors-...
https://www.digitaltrends.com/mobile/apple-siri-listening-re...
In fact, you do. Amazon Echos have a physical button that disables the microphone. Apple HomePod has a setting to disable the "Hey Siri" feature. I haven't tested this, but I presume that both continue functioning perfectly fine as Bluetooth/AirPlay speakers.