It is looking away that breaks conversations, not doing something else.
I am not disputing their unquestionable usefulness I just have doubts they will be acceptable in most social situations.
If any of you are wearing glasses, try looking at the top corner of your frame; that's the same motion needed to engage with Glass.
You'd effectively completely break eye contact with the person you're talking to and motion as if you're looking at something completely different. It's the equivalent of getting distracted by a cool car while your friend talks to you and you have to say "sorry, I saw a cool car, go on".
The thing I think will be weird is that Glass is in the upper right, and looking up and right is associated with creativity aka lying. Not sure why they didn't go with upper left, which is more associated with memory recall.
Both these eye movements happen in normal conversation btw.