Non-native speaker here. Can't think of any rude ones, but here are some odd ones:

"Sperm whale" always sounds a bit funny to me, but I don't know whether native speakers find it funny, too.

"You can't have your cake and eat it, too." I understand the underlying logic, but it always seems a bit weird, since 1). you can eat half a cake; 2) if you don't eat a cake, you will no longer have a cake in a few days anyway; 3) it's a very long phrase for a short idea; and 4) the fact that "have" can also mean "eat".

"blow job": seems to be the opposite of what the job entails.

"first off": Why the "off"?

"ladyfinger": I understand the finger part, but "lady" finger? A bit cannibalistic, isn't it?