The only concern I've seen is people who might have reason to keep a phone's existence secret. Those with abusive partners, parents, etc.
Here's a bit more information for you, it should help with the fear-mongering bs.
> https://www.reuters.com/fact-check/out-of-coverage-switched-...
It's called a roving bug and dates back to at least the mid 2000's https://www.cnet.com/news/privacy/fbi-taps-cell-phone-mic-as...