You can not be held in contempt for invoking your 5th amendment rights when they apply. That's one of the things the fifth amendment protects against.
Like you cannot walk into a bank with a gun yelling 5th amendment and not go to jail. It's not blanket. It protects you from statements of self incrimination.