For instance, at a meeting 20 people have to wait for that one guy to get on the phone. Or now you have to pass the Klingon mic around so that the remote worker would hear properly. And god help you, if the remote folks are in a different time zone - now 90% of the people in the office get inconvenienced because someone is 3 hours ahead of us.
We have this very situation at work and it's very unhealthy and annoying. I've come to conclusion that on large teams it's either everyone is remote or no one is.
Hanselman (http://www.hanselman.com) has a good series on working remotely and making it work - he outlines the problems and how to solve them. At least he is trying - other folks simply do not.