They only miss out on this if the people they are working with are in the same office as them. This isn't always the case at large companies.
I suspect that people who work for companies where teams, organizations, or even the entire company are colocated will tend to favor in-office work to some degree, while people who work for companies where teams and offices are geographically distributed will tend to favor remote work.
I personally worked for Twitter in-office full time for 14 months pre-COVID, and almost every meeting or collaboration I had involved a teleconference with people in another timezone or was done over Slack. There was some opportunity for face-to-face collaboration with people from other teams from time to time, but in my opinion not enough to justify the daily commute. That said, I'm open to a hybrid arrangement for companies where the situation I had at Twitter would not be the case.