One can at least assume that most people carrying phones don't have their microphone and camera on all the time - with Glass, that intrusion into privacy is designed to be unavoidable.
The only real difference is positioning, but you can hold a phone up, and most people won't notice.