I spent quite some time with the Swedish retroflex fricative, and always heard something else than my teacher described. I finally settled on learning it by IPA, i.e. learning where exactly the tongue needs to be.
Even harder are Russian consonants for me: I can try as much as I want, but I will probably never be able to distinguish Russian hard and soft consonants, even with a Russian speaker saying them slowly and in contrast (and with lots of "you're kidding me, right? Those sounds are nothing alike").