I find your comment amusing because, of course you can. And famously, the people that made it big did just that.
But I get it, I'm like you. Also had a college loan to pay off, later kids.... So I stuck to the day-job, read about the high-flyers and work instead on trying to get-rich-slowly.