I used to do that.

The difficulty with this state of affairs is that the work you do at your job begins to become a [time+motivation+energy] sink, and you often find yourself wishing that you could devote more of yourself to your pet projects.

This in turn affects your performance at work, giving you a perpetually negative loop that's broken when you shift jobs.

There are very few excuses, I think, to not find good work these days. Pay with personal circumstances is the only I can think of - if you have a family with enough children to make accepting lower pay for better an impossible decision, for example.

I beleive that you have just this life to live. Make the most of it.