There are some number of jobs out there and most of them don't want remote workers. Job applicants that are willing to take jobs that forbid remote work can also apply to those that permit remote work, while workers that will only look at jobs that allow remote work can only apply to the small subset that allow it.
It's straightforward micro-economic exercise to see that the when you hold all other things equal the market clearing rate for a particular employee is higher if he is willing to take on-site work than if he is not.
It'd be the same thing if an employee was only willing to work at companies whose names started with a-m.