It's not about their company and the building they reside in. Wealthy people operate as a hive mind. A network. A dive in commercial real estate is a hive problem. They're all panicking for each other because they affect each other.
Something I note from interviews from RTO advocates is that the managerial class has an emotional reaction to WFH. I believe it wounds their sense of status and hierarchy - they resent the "unearned" freedom and comfort of people they have power over. They are not acting for business advantage but from a compulsion for their inferiors to know their place. It's very human sadly.
I don't know if this is actually true or not, but it is exactly how I feel (about the mangerial class).
They have the same interests, hang out at the same places, etc. They need not conspire, it's automatic