Do you move the connector while it’s plugged in e.g. if you have the phone in your pocket plugged in and then connected to wired headphones? The lightning connector works fine when you don’t move it, but not when you do.
But really what I don't understand is how Apple can't just add an option to turn off the damn accessibility options entirely. Why does dirt or temporary bad connections cause this problem at all? They could just let you turn off the setting that it triggers so that it really doesn't matter. The fact that they don't care enough to do that tells me enough of how much they care about making their headphone adapter a good product. I never used to have to worry about issues like this with a regular headphone jack, but I certainly do with Apple's adapter. It's simply a poor product.