I'm at a party, and a friend is demonstrating his juggling skills. I could snap a quick picture or take a video in a heartbeat if I had Glass. Since I don't, I have to go through the smartphone activation checklist again; take phone out of pocket, power button, swipe. Launch the camera app. Aim. But now I've missed the moment. I'm holding a brick in the air while the rest of my friends are applauding.
Damn, my car won't start - and unlike my dad, I've never been much of a mechanic. If I had Glass, I could give him a call and he could see through my eyes from 1000 miles away. Instead I'm cradling my phone on my shoulder while I'm leaning into the engine, looking for a relay box. I don't know what that is, or what it looks like. He says it should be somewhere on the left? Now I know how he feels when I'm guiding him through installing printer drivers.
I see a lot of use cases for a device like Glass. So I think you have it backwards - in the near future, the dorks are going to be the ones fumbling with their phones.