I find I'm most productive in the morning so I try to dedicate the first hour to two hours of my day to my side project.
So I would try boxing time early in the day before your full time job, focus on your side project then. Work your full time job come home for family time, get a good nights sleep.
Occasionally I will work on it late in the evening if the family is out and I'm home, and sometimes on the weekend. I just find it easier to get things done early before everyone is up and going about their day.
Giving up TV shows and Video Games can give you extra time for your side project. Although everyone needs some down time so I wouldn't say you need to give those up completely if you enjoy them.