I visited Fog Creek in NYC once and as you know Joel Spolsky is a very big advocate on giving devs a good work environment. However he is also a big advocate of hiring good people no matter where they live. They had a policy that they had to discuss everything in chat or teleconference from what I saw. They said that despite being a door or 2 down from someone they wanted to talk to the forum of discussion had to be online for the sake of the remotes too.

If it isn't like this (and most places aren't like that) you are going to miss out on things when not in the office.