Isn't the context thing true for anyone learning a second language?

I was an immigrant in the US as a teenager and it felt like that in English for me my first few years here. Eventually I got to a point where I can understand stuff without actively paying attention, and it got a lot easier to understand anything with or without context.

As an interesting aside, I speak Cantonese natively, and I barely speak Mandarin. Mandarin to me is like that as well. I have to pay 100% attention to keep up, and as soon as I lose context, all hope is lost in understanding anything in the conversation forward.