You don't have to buy Apple's dongle. I found third party ones for $1.46.

At the price you could just buy like 5 and permanently attach them to each pair of headphones.