I didn’t understand English that well when I was a kid but since we don’t translate English language media, except for subtitles, I got to hear English over and over again and now my ability to comprehend spoken English is very good. As is most Swedes, we spend just as many hours a week in English class as they do in France, Spain or Germany yet we are better at it (in general). We’re not geniuses, we just get more practice and less hand holding.
Can’t do anything about books though, sadly. If I read a book in a language I don’t understand I probably have the intention to learn that language.