From having tried Glass, I can affirm that you do indeed look away.

If any of you are wearing glasses, try looking at the top corner of your frame; that's the same motion needed to engage with Glass.

You'd effectively completely break eye contact with the person you're talking to and motion as if you're looking at something completely different. It's the equivalent of getting distracted by a cool car while your friend talks to you and you have to say "sorry, I saw a cool car, go on".