If you live more than 30 minutes away from the office, and it's self-evident that your physical presence isn't necessary to accomplish your day to day, I don't see why someone wouldn't refuse to come back excepting a few days here and there for a fun communal activity with your team. Especially if they have children.