And therein lies the rub.
I'm also reminded of the studies that document that routinely exceeding a 40 hour week causes reduced productivity (IGDA "Why Crunch Mode Doesn't Work"), and that 40 hours was based on assembly-line work and the corresponding number for creative work is probably 35 hours max leads me to think it's just not possible to have a side project without interfering with your main job to some degree.
So yes, I've gladly given up TV and movies and gaming, but that still doesn't seem to make a side project feasible, which is really sad.