The real downsides for me are: 1. The occasional loneliness 2. difficulty in casual collaboration with colleagues 3. the "work hangover" where after finishing work, I don't have adequate physical separation from it (like leaving the office gives you), and sometimes I can waste time after work is over before being able to spin up on something else.
The upsides are: 1. Music when I want it, silence when I don't. 2. Cheaper food costs, healthier meals 3. no commute costs 4. more overall time to indulge in hobbies/side projects 5. tax benefit of maintaining a home office. 6. save on dry cleaning, clothing costs 7. can run errands during the day, and always have the 2-5pm block of time while normal companies are still open, and I am free.