I spent three years in Central America, mostly hanging out around gringos, and six months immersed in the language. The difference was incomparable. I don't consider myself fluent, but I think I can say with some authority that the more exposure to native speakers that you have, and the more that you force yourself to express yourself in that language, the easier it will be to learn. There aren't many people for whom learning languages is actually easy, but there are easier and harder ways to go about it. The important point is that it is something that is possible for anyone, with sufficient effort: It's wired into our brains.
As an aside, I would like to recommend the app Duolingo. I haven't used it much; it was a little embarrassing to be using it around a bunch of native speakers, but I do think it helps to gamify the experience.