Going freelance isn't in itself a poor decision. Quitting a paying job without a steady stream of work already lined up is.

Start trying to build a customer base on the side. Once they're bringing in enough to cover your expenses with a reasonably healthy percentage over that, then consider quitting your regular job.

Freelance work is not steady work, so you have to be sure that you can cover your expenses with regular work before you should be thinking about dropping a regular income for it.

You will need to build up a steady base of regular clients, and build up word of mouth. If you're not a designer, then it becomes really hard for people to discover you in any way other than word of mouth.

tl;dr: Don't quit your day job. Work extra hours until you're earning enough regularly and reliably to cover your expenses, then consider dropping your full-time job.