There is already a social stigma around people being focused on their phones/screens, and the disconnect it creates with people around you. But that disconnect is somewhat virtual in that the phone is absorbing your attention, but it's not a physical barrier.
These AR glasses all, by design, create a very real physical barrier between the wearer and their surroundings. It also doesn't help that they look goofy, and sometimes slightly creepy. I think this creates a much bigger barrier to widespread adoption, it is a much larger conscious decision by a user to wear an AR kit. You can't just glance at it and get sucked into the content stream, it is a very distinct process to put it on/take it off.
I've been intrigued by the Apple Vision, and other AR devices, but it still feels like there isn't a breakthrough moment for them.