I think what's going on here is that the context of the sonic character of an individual's voice is crucial to the brain's processing of the phonetic content of utterances.

I spend my whole life listening critically to things. I was 100% YANNY on two sets of headphones and signal amplification paths. When I moved the slider bar about 1.5 notches to the left, LAUREL sort of "came into focus", and I felt like I grasped the human voice behind this severely distorted recording. Then, when I moved the slider back to the neutral position, I could not hear YANNY at all anymore. Only by taking a brief pause could I 'reset' my brain to hearing YANNY.

There's another processing step going on that helps us deal with distorted or band-limited voices. We're reconstructing in our heads the ideal sonic image of the human voice we're supposed to be hearing.

I hypothesize that if the sample in question came at the end of a few words by the speaker, not related semantically, e.g. "right, seven, purple, elephant, laurel", so the listener could reconstruct a more complete "imprint" of the speaker's voice in their mind, the YANNY outcome would drop to <1%.