Other than being single, I'm in the exact same position as you. I'm pulled between building a career and travelling the world.

I graduated last spring and spent the summer backpacking in Asia before starting a boring desk job. I live with my parents still so I'm saving almost all of my income.

Now I'm seriously considering quitting in six months and heading back to Asia until I figure out what I want to do. My job is terrible, especially since I was effectively lied to about what I'd be doing, but I'm more or less getting paid to do nothing. In six months I'll have enough money to spend a year backpacking across SE Asia, Eastern Europe, South America, wherever.

Or I could just quit now and go find a job I like. I'm honestly not sure what to do.