Especially a mix in which certain team members are perpetually "outside" the main group. The remote participants simply don't have access to the same side-channel discussions office residents do. Location, bandwidth, and multi-channel communications (sight, sound, and even touch) matter.
The places I've seen this work well (very rare) involve people who travel frequently between locations (expensive and personally disruptive), and in which work and participation are forced onto the same channels remote workers use (chat/IRC/voice).
And sadly, in 2011, speakerphones and audio conferencing still sucks massively.
If you've got a very strong, very cohesive, very long-lived team, with a history of strong direct interpersonal relations, you may be able to get away from this.
Another factor I suspect has a strong bearing on this is a team in which members are chosen by merit (or at least on a broad-based acclaim, not by a single hire/fire entity), and who cannot be easily shed. Scientific communities, some forms of government collaboration, and much of the Free Software community operates on this basis.
Knowing your job is NOT at stake for any given mistake, error, or political misstep you make does a great deal for such activities to engender stronger and better participation.
In my experience.
I do some work at a large university where they have abundant bandwidth and the latest "telepresence" hardware, and as often as not you are still dealing with laggy audio/images, freezes, dropouts, echos, individual participants having technical issues or ignorant of how to use the systems, etc. It just doesn't work smoothly enough to be a pleasant experience, and often times is such a distraction as to render the entire exercise a waste of time.
One time, it was 2am and my friend and I had just been kicked out of a bar near our office. There was also a severe thunderstorm warning in effect (with a lot of lightning) so we decided to go back to work instead of riding our bikes home in a tornado. Being rather inebriated, we did not do much work. Instead, we wandered around the building and found the telepresence rooms. We wondered, "how do we link these up and have a conference". There was a phone in each room, and they had some digits on them, so I went to the other telepresence and punched those digits into the phone-like device there. Connected! We then had a conversation for a good hour or so; it was so much like real life that it wasn't worth the effort to get up and walk back to the other room to have a "real" "in-person" conversation. There was no latency. Eye-contact was eye-contact. Everything was the right size. The displays may have even been Retina Displays; I didn't notice any pixels like I might when watching HDTV.
So all in all, not bad when done right. It passed the "wait out a tornado while drunk test", which I think says a lot.
So: being right next door was practically like being ... right next door.
Just as "open floorplan", hot-desked, minimal personal space offices make lousy working environments for technical workers, a glass, steel, and dry-walled office with no acoustic absorbing materials, poorly distributed mikes, wide range of speakers with varying levels of accent and enunciation (one Russian I'd worked with was the black hole of speech, even in person words were unintelligible beyond an event horizon of approximately 12 inches), and the usual technical glitches (digital distortion, dropouts, varying volume levels from multiple remote locations, ambient noise locally or remotely ...) and the experience is uniformly barely tolerable.
There's a reason any professional AV situation (panel, conference, broadcast session) has one or more sound engineers. People may not appreciate the complexity, but it's a non-trivial problem.
Yes, it's cheaper than flying a bunch of people all over the planet (a lot cheaper), but "cheaper" != "good experience". And when the result is that the time is in fact wasted as you say, I'd say it's worse than the alternative (no meeting, written communications, personal travel).
And casual, informal communication is a major factor in teamwork.