Then don't work from home?
I've got a flexible work schedule, "working from home" can mean many things for me:
-Work from home in the morning, then stop in the office in the afternoon (my commute is short).
-Work from a coffee shop, beach, etc.
-Meet up friends for the day while we do parallel-work.
-Head out of town for a 3-4 day weekend and work remotely for a couple days without having to use any vacation.
Also worth noting, my co-workers with kids get even more value out of this flexibility than me!