Thus RTO improves group A performance while not affecting group B performance so it looks like a win overall and in a sense it is.
But it is penalizing group B, making their lives inconvenient, and increasing their expenditures.
Working from home is objectively better if your employees are in group B: (1) better for the environment [less transit], (2) leads to better work-life balance [no time lost to commute, more flexible work hours], (3) is more cost efficient [no need to rent offices in high cost areas, remote employees can be cheaper as their COL is often lower], (4) lets employees have better homes [because you don't need to be near the downtown core].
I think if you start remote and you measure performance of your employees, you can ensure that you get most group B employees and you can stay remote. The issue is if you started in the office, you have a mix of group B and group A, and RTO will improve the average employee performance, so it makes sense.