Even without RTO, people will still leave for other jobs if the grass is greener on the other side. If anything, WFH makes it easier for someone to go job hunting and do interviews.
If the environment is your downside to RTO - you’re reaching.
The loss in productivity when someone does 2 hours in traffic a day is probably a more compelling argument. Companies also have to stock their pantry, hire more office management, renew their office space leases.