Offices are designed to keep you there and focused on the work as much as possible. At home I have my pets, wife (when she's not at work too), things I like, in a location I chose to live. People call these distractions but they're your life at least as much as your job is. At some point, working at home, I'm going to stop and pet my cats, go for a walk with my wife, sip coffee and read a book, visit a friend, whatever. When I have kids, I will get to be there and see them grow up.
When you're able to get a good amount of great work done because you have that uninterrupted time, it makes the idea of working 8 hours 'just because' seem stupid and opposing your productivity and your happiness. I get more and better work done these days. I make more money, for both myself and my clients, working from home, and I much more rarely need sick/mental health days because I'm basically happy and healthy all the time as a result of working from home.
The things I miss, like socializing with a team, I can get other places. I still talk to other devs during the day, just on the phone, over video chat, or something like slack. I go to meetups dedicated to the things I'm interested in, chalk full of smart and interesting people. I have friends to socialize with, many of whom are from those meetups, so I get to talk about tech even more if I want to.
Basically, the downsides for the employee/worker are heavily, heavily outweighed by the benefits in my opinion. I also think that's the case for companies as well, but only have my own small experience as a very small employer to back that up so maybe someone else could comment there.