If I am disrupted 5 times with 5 small things in an hour, that will slow me down significantly more than if I am disrupted by one big thing in that same hour.
For a tech analogy (we're on HN after all), being in the office reduces latency but in doing so can also reduce throughput.
For the kind of work managers do latency can be critical, so from their perspective being in the office is extremely helpful. Other can be most productive when given large amounts of time to focus on a single task in which case latency is far less critical.