This rapid and random access is why people love smartphones so much. With a headset, you either have to be wearing it all the time (something people aren’t willing to do; it’s like wearing sunglasses indoors no matter how good the waveguides are) or have to take the time to take a fragile headset out of its case, put it on your head, adjust to the complete take over of your field of view & environment, then figure out what you need to do using an input modality (be it voice, gestures, an input peripheral or some combination) that is far inferior to a touch interface for the 90% of tasks.
People like to claim that AR is less intrusive than a smartphone but all the cases they present are carefully crafted demo’s. If you watch actual people use AR devices trying to accomplish some task you immediately notice that isn’t true at all.
It will be interesting to see how Apple has addressed these problems but AR needs to be better at doing tasks average people can already do well with a smartphone, and like I mentioned you have an uphill battle due to the headset form factor alone.