That's just a Bluetooth capacity problem. Bluetooth isn't built for high throughput scenarios and "HD bidirectional audio" is considered high throughput in this case.
Same problem happens with a combination of earbuds and a smart watch, or headphones and a Bluetooth mouse, depending on the interference and chattiness of your devices.