You are right that the police can't force you to talk and that a judge can't if you are on trial.

However a judge could force you to testify in someone else's case. You can't broadly plead the 5th in that case unless the testimony is incriminating. Let's say you saw a murder happen at a hotel you were at while cheating with your wife. A judge is allowed to force you to answer the question.