As you listen to someone, your brain is constantly matching the sounds arriving at your ears with a prediction of what the next few words might be. Listening in a non-native language, my predictions about what comes next aren’t very well tuned at all, so if I can’t hear every word clearly then I can easily get lost.
Another signpost: sometimes you mishear someone — “oh, I thought you said xyz” — but the thing you thought you heard them say is never gibberish, it’s a grammatically and contextually valid way to complete the sentence.