> But why does it have to be an either or?

In mixed onsite / remote teams, the remote employees can be seriously disadvantaged regarding access to information, participation in decision-making, etc.

Some of the comments above mention this effect, and my own personal experience has very painfully confirmed it. It was a truly awful experience for me. I suspect it's what lead to the maxim, "If anyone is remote, then everyone must be remote."