For me it's areas a few blocks around Times Square. The only thought I had (and maybe the Oculus is the same) is that the sheer number of people also using Airpods in the area means everything gets confused.
In short, it's the combination of no walls + lots of signal congestion. Times Square + Oculus both fit this mold.