I think you are describing this behavior: https://support.apple.com/guide/iphone/switch-airpods-automa...

You can turn it off by doing this:

"To turn off automatic switching, go to Settings > Bluetooth, tap the Actions Available button next to the name of your AirPods, tap Connect to This iPhone, then tap When Last Connected to This iPhone."