I'd contend that the tools just aren't there.

When I want to chat with someone, I want to do it instantly, like walking up to someone in the office. The Skype concept of "dialing" someone is old, antiquated and feels... laggy.

There was a video chat client promoted on Hacker News a while back that allowed for instantly initiating a video chat session with a colleague.

Sococo, a distributed agile client (with video support) also claims to do the same thing, allows for you to instantly create a video session with anyone in your virtual office.

IM works because it's immediate. Video chatting must be the same in order for a distributed team to work.

The tools aren't there yet.