What about using a different microphone? In my experience, listeners prefer the built-in MacBook Pro microphone over the AirPods Pro microphones. Or if you have a USB microphone, you can try that as well.
It's mostly fine, but I also step away from my desk. I wish I could also prioritize my microphone.
1. Webcam (if I'm at my desk, I have my webcam) 2. MacBook Pro (if I'm away from my desk, my computer is still probably within range of my MX mouse)