Managers, especially has you become further abstracted from providing real value, depend on "face time". It's their currency. Remote work really hampers that. A big meeting with the boss on Teams is going to have less chit chat and really no opportunity have side conversations or get 'face time' before and after the meeting.
I think remote work is also hard for bad managers and good for good managers. Bad managers care about time rather than value and rarely understand what's actually going on. If they don't understand the value their team is creating and they don't know how much time their team is spending sitting on a computer, they really don't know anything.
Good managers on the other hand understand the value the team produces and don't really care how people spend their 40 hours, so I doubt they sweat it as much.
Pretty much all the concern from management is veiled ways of saying 'this will make it harder for me to rise in the organization as I can't build personal relationships as effectively'