Putting Amazon/Apple's privacy and security implementations aside, how are these different? Aren't they both 'always listening'?
If you are using a voice assistant on your phone, and if you have it set up to always listen, then it's almost the same as a smart speaker (minor quibbles about the positioning and quality of the microphone aside). But those are two pretty big ifs.
Usually this argument gets pulled out to shut down people who have concerns about smart speakers in general, saying that any effort to make smart speakers more private or avoid them is pointless because of course their phone is always listening to them. That's not necessarily true. If someone wants to get rid of their smart speaker, owning a phone doesn't immediately make them a hypocrite. They can keep their phone and still have better privacy.
You're correlating risks that aren't related to each other; these are two different devices with different threat models. If the NSA wants to bug you, it will bug your house. That doesn't mean you should bug it for them.