(Tangent, but I’ve never found this particular example compelling. All that’s in my underwear drawer is underwear... and I don’t find that particularly embarrassing? I think I’m missing some cultural context for it)
In the US, at least, the underwear drawer is a popular place to hide other things ... eg, your illegal drug stash, or the engagement ring you bought, etc. I think there's also subtextual implications of nosing about in your sex life.
It's a mild, nonspecific stand-in for whatever it is that you really don't want people knowing. Whether that's your weird fetish or your embarrassing hobby or the fact that you worked as a pianist in a brothel or whatever.