I currently work remotely and enjoy it for a lot of the reasons you say. However I've had a pretty tough time with loneliness. I moved to a new city (relationship reasons) which is always challenging, but I think remote work has made those challenges particularly acute.
I think of myself as a fairly social person (at least relative to software engineers) but I've struggled a bit with this one.