Is that actually the case?
I've always found it easier to follow a conversation in a language I am learning than to speak it.
You get an awful lot of context when listening, only need to put a few words in the right place and suddenly what you're hearing makes sense. You can get away without literally translating everything.
Speaking on the other hand, you can't converse properly without being able to find the right word at the right time and in the right place.