You conveniently missed the first part where I said their output has noticeably dropped during the remote working period.
You sound like a micromanager. If you have notable metrics that show decreased output, then raise the issue and correct it. But don't say that everyone should be subject to losing privileges due to your team's poor performance.