Maybe that's why so many companies push for people to return to the office. In-person working where someone can just walk up and interrupt someone else is equally awful for everyone.
Aside managers like people in the office because they are more tied to the company, change employer often means having to move elsewhere and the worker might be in couple and his/her partner might have issues moving, they might have kids suffering from loosing friends and change schools etc, this means workers more keen to accept bad condition at work at a certain point in time. A remote worker can potentially change employer just changing some login screens.
But in most companies, it's both.
And people suck at discipline.
I don't think such scapegoating will actually help the remote crowd.