I'm a Polish expat in English speaking country and have the same experience.
When I screw something up at work I just swear in Polish as it's not as obvious to people around me as it would be had I used a profanity in English.
Also, love to listen to heated, swearing-heavy exchanges from Italians or Spaniards. Don't understand a word but it sounds awesome :)