I often hear people use the phrase "call a spade a spade", thinking it's saying "if a card shows a spade, call it as a spade (when showing the hand)", but I've heard for a long time that phrase is also racist. Not growing up around anyone who ever used that phrase, I would have had no idea.
Makes me wonder how many other phrases out there in every day use don't have a happy meaning, even though at first glance they might seem innocuous.