Putting on your headphones at the office and trying to get work done is seen as antisocial and hostile (especially daring to direct people to try to get you on im or email), and you find yourself shut out of critical decision points meetings because people will say "we didn't want to bother you" (what they're really saying is "fuck you, thats what you get for isolating yourself").
People want to be able to get immediate responses and interrupt each other whenever they like, and its easy to be pegged as an "asshole" if you don't play along. If you extrapolate from there, it starts to make sense why remote work is still such a hard sell in a lot of places (last stat I saw is that only about 4% of American workers work from home, thats up from 2% last year, but still)
"Hell is other People" --- Jean Paul Sartre