I'm a native (British) English speaker, and I actually can find it quite hard to communicate with non-native speakers, because they don't understand my accent. It's not particularly strong as British accents go (nothing like Glaswegian or Geordie for example), but it really caused problems when I moved to a research group which was made up of primarily English-as-a-second-language speakers. The other thing I found was that while their speaking ability was good, their written grammar was generally poor; I'm the resident proof reader in each of the places I've worked.
Many of the people spoke what is apparently described as 'Euro-English' which has many borrowed phrases from other languages, or what native English speakers would see as mistakes but which are relatively easy to understand between non-native speakers.