Productivity is not the be-all end-all of a career! Human interaction (of any kind) is important, "wasting" time going to lunch with colleagues is important, serendipitous encounters at the coffee machine are important.
I know for certain if my employer decides to go permanent remote, I'll be looking for another employer. This is not the way I want to live and work.