This is exactly the reason why I disabled it when I upgraded to my current M1, and I'm relieved to hear I'm not the only one experiencing this.