Speaking is harder than listening because it requires you to produce language and convey ideas from your own mind, with proficiency in pronouncing it in such a way that a native speaker might understand you. Things like idioms you may be able to infer meaning from interpretation when you are listening or reading but you will never use them in speaking unless you really truly know them. You’re correct that the variables of listening are out of your control (which is what makes it so difficult as compared to reading), but that does not make it harder than speaking.
In the military where the DLPT actually tests all three categories for proficiency, speaking is consistently and universally, regardless of target language, the one language learners are least proficient at.