I do that, I'm not the owner either. My wife just had a son, so I definitely have a family. Add on top an hour and a half commute each way and it can be a little difficult at times to have family time I'll admit.
Some of us don't have a choice but to work jobs that require that many hours to pay the bills.
I doubt your spending any more than 30 mins per day with your son on weekdays.
My day off is all about family, so it's daily planned :-)
Too late to edit haha.
(Unless you're doing this just for a very limited time for a very good reason. But don't let it become a lifestyle!)