They're points worth making, but not really relevant. All I want to know is, if I answer a call and listen, and don't say anything, is there a chance my device could be compromised?
Very unlikely. If you're not worried about social engineering then I don't think you should worry about it. Only thing is that sometimes if you do pick up you can be more likely to get those calls, since they know you pick up.
If you were thoroughly paranoid, you could make a case that the background noise might, at times, give hints enough for a clever man to roughly figure out where you are.