You think you can do it with just a few years of learning, but talking in English to someone you share a language with is entirely different than talking to someone of a very distant origin. I remember trying to do it with my classmates in our very similar accents, with our common lexicon, as opposed to talking to Asian people during visits abroad (I am European).
Even with fellow Europeans from a different country it's not easy to get the message across, even between similar languages like Spanish and Italian, to the point that if you're from Spain and need to get something in an Italian shop, you're probably better off by speaking Spanish, statistically it will be understood much easier than English.
To sum up, English is not easy at all. It's just a bit easier and surely worth the effort given the great amount of resources already available in English, including the giant pool of speakers.