Reasons against remote working: 1. Timezone issues - this is, imo, the biggest one. Especially after working with people in China, India, Europe or even eastern USA (with me in SF bay area). People all over the world usually start working around 9-10, and stop working around 6-7. Meetings, code/design discussions are very difficult on those schedules, especially when you want a decision on smaller issues where a quick back n forth would settle the matter in 30 min if everyone was in the same room. And there are plenty of such issues, where unavailability of your coworkers seriously affects the productivity.

2. Rapport / Camaraderie: Knowing someone in-person, having lunch/dinner/coffee with them, having watercooler talks about casual stuff etc builds a rapport, which helps build closer work relationships. I don't think people in remote locations are any different than my colleagues here, but the rapport is simply not there if I am going to meet them on video conf or IM only a few times a week.

3. Impromptu discussions: These are triggered by spontaneous ideas, and ideas do not come according to a fixed schedule of meetings. By not having remote workers in these discussions, both sides lose (by not having more brains, as well as by missing out).