There's multiple effects going on. Firstly the bluetooth spec itself is quite complex: the 'core specification' document is near 3000 pages of very dense technical details, because it's aiming to allow anything to connect to anything. Secondly, many bluetooth devices are both heavily power and cost optimised: you don't have a lot of silicon and you don't get to use it very often if you want your product to have a reasonable battery life. This is especially true of the raio in a bluetooth device, and adds a whole other level of complexity to your implementation, and is one area where bugs are very easy to introduce. It also requires a fairly complex dance between hardware and software (most of the bluetooth spec is implemented in software, usually C. The hardware mostly consists of a radio capable of transmitting on the right 2.4Ghz signals with the right modulation scheme, but when you add power management to that it gets gnarly).
Thirdly, hardware companies are generally not great at software. A lot of effort goes into physical testing and validation but the base level of quality of the code isn't great. Also the quality of a given bluetooth device often depends on the quality of the custom code for it: there's a generaly trend of running application code on the same CPU as the bluetooth code, and in an embedded context there isn't generally great seperation between the two (nordic's approach is probably the best here). A third-party making a cheap pair of bluetooth headphones is not likely to make a quality result, even if using a library which is relatively bug-free.
Finally, there's a vicous sprial effect: because you need to interact with a wide variety of other implementations, which may all be buggy or quirky in their own ways, you wind up needing to test with a wide array of other devices (expensive and may not always happen) and play whack-a-mole with the remaining issues which appear, which in turn creates more quirks and more opportunity for bugs.