Oh no, I didn't mean any of those at all. Quite the opposite.
I mean "understand it" as the opposite of "it sounds like nonsense." It's at the intuitive level, not about the ability to reason the structure details or whatever.
Everyone grows up listening to some particular types of music in their household, culture, etc. You naturally develop an intuitive understanding, in the same way a native speaker has adopted the rhythms and tones and phonemes of their local language. A typical modern American listening to Charlie Parker rip through a bebop solo, is not too different than listening to someone speaking Mandarin. The sounds don't map onto anything recognizable. But you listen to it enough, and the brain starts to understand the structure. It's not just a blurred stream of saxophone, but a series of phrases, with inflections, patterns, meaning. Listen to it long enough, and you hear every individual note as it flies by.
Not necessary to read the score. But a familiarity with the structural conventions helps the listener contextualize the listening experience. If I listen to Mozart or Haydn, I should expect that I’m going to hear something in sonata form, meaning melodic exposition, thematic development and recapitulation, with some variations around that format. If I hear a Bach fugue, I’m going to hear even more formalized structure, a subject (ie theme) and then restatement of the theme (initially a perfect 5th lower) then introduction in other voices and elaboration.
Maybe there’s even a wider context to place it in. History, how the structural and other elements mirror similar characteristics in the visual arts, architecture etc.
TBH I listened to (and attended lives performances of) what must be hundreds of hours of Mozart, Haydn, Beethoven string quartets and such, before I finally bothered to look up "sonata form" not too long ago. I can't say the lack of knowledge about the exact [:Expo:][:Devt+Recap:] structure impacted my enjoyment or appreciation of it. I mean, one quickly picks up on the repeated motifs and the "now back to the beginning" patterns without specifically picking out the expo/dev/recap sections.