Most remote workers will also tell you they do not hate on-site. Being at the office, meeting co-workers is great for mental health.
What's not great is schools end in the afternoon, young children can't be left alone at home. What about a sick kid at home? These are real problems that our society has washed it's hands off expecting people to 'figure it out'.
Also spending an hour in commute is no fun, more so if you are a family person.