Agreed that quality audio is key. But poor, stuttering video can really be distracting. In Locus we will typically just stop a video stream before allowing the stuttering experience.
Our brains are familiar with a phone call; it's just not a problem in most cases.
The remaining cases are probably more like meetings with unfamiliar people, or webcam sex chat.