You haven't mentioned how your side project is doing. Have you seen reached any milestones/small successes? Is it something you can monetize, like a SaaS website (as staunch wrote below) or perhaps an iPhone app? If you are already spending some free time working on a side project, you're on a great track. What's actually hard is to allocate time to this every single day, since as you noticed, other important things come in the way. A couple of hours here and there likely won't do it, but this doesn't mean you have to go "cold turkey" yet.

In your position, I wouldn't recommend quitting the day job just yet until you have some financial buffer, so you'd need to figure out a revenue stream first. You don't have to passively wait until things get better, but instead focus on shipping some things and perfecting your art.

Finally, your family (and especially your spouse) has to be on board with this, and it will help you immensely. If you're spending those few hours on a computer, they should know that you're working on something and not, say, reading the news. It'll take some time (or achievements) for them to take it seriously, but, again, this will really help you.

Best of luck to you, and don't feel trapped. You have to be genuinely excited and know that you're doing the right thing every day that'll take you to where you want to be.