Even dedicated conferencing rooms have poor quality mic setup (easy to fix) and acoustics (a little harder)
It's testament to the ability of echo cancellation and associated algorithms that these systems are able to make anything work at all -- and in many ways this has masked the problem, preventing attention where it's needed.
I develop a 'conferencing' system for radio broadcasters called cleanfeed.net. It's amazingly basic compared to more general systems, (it basically doesn't do any changes to the audio) and yet the rapport is much greater, even with smart use of basic kit -- you can chat over the top of someone, interject and conversation 'flows'; no video needed.
Of course, this is very much a solution for a certain kind of "advanced" user, but I believe the solution is somewhere in the middle -- too much attention spent on video (not audio) and on software (rather than provisioning the right hardware)