From dating a Polish girl, and meeting some of her relatives and friends in Poland, I'd say it's mostly a problem of confidence. Once they started talking there was rarely a problem (at least for non-rare words), but a lot of them didn't speak English to non-Poles often enough to feel like they actually could.