Correct me if I'm wrong, but wasn't NFC at one point considered the solution for such out of band pairing? I think NFC headphones are still available for sale.
You'd go up to the speaker (which you had to do anyway to turn it on), and you'd touch the phone to the NFC part. That would turn it on and pair it with this specific phone. The whole thing took less than a second.
It was great for sharing the speaker among family members, when different people used it at different times, each with their own phone.
This was in ~2015, I had a Galaxy S4 at the time, no idea whether this works with iOS or modern Android.
The spec. allowed to exchange encryption keys with a different method than Bluetooth, Sony is using it on the Playstation to perform BT-pairing via USB.
Commercially, NFC was mostly used to initiate pairing, by having a NFC Tag on the accessory which stored the Bluetooth address, and a device scanning the tag would initiate pairing with the device directly.
The pairing itself is technically still done over Bluetooth, which is nowadays mostly a matter of confirming the operation...
/s