The problem is many of the remote employees were the truly valuable ones. Talented engineers get spooked pretty quickly and a flexible work schedule is a very good reason to stick it out at a struggling large company. The more of those reasons you take away the more you'll find yourself with less desirable engineers.