It's not just that nobody cares enough to struggle through with you, but people actively will try to speak English with you as a way to get more proficient in English.
(There are a lot of expats among the parents at my children's school and it's easy to tell who persisted and who didn't.)