Wouldn't that be a reasonable time to do this?
Disclaimer: IANAL.
EDIT: I'm getting some downvotes, so I will link to this that seems to support what I'm saying:
http://criminal-law.freeadvice.com/criminal-law/white_collar...
Perhaps my use of 'falsely' is confusing, but what I mean is if what you are refusing to say is not actually a crime, and it is discovered, you could be charged with a crime (i.e. perjury, giving false statements under oath).
It seems like you're approaching the 5th amendment as a direct point/counterpoint arrangement: the police accuse you of robbing a bank, you know you didn't because you were cheating on your spouse, you can't take the 5th because cheating on your spouse isn't a crime, so you must answer "I couldn't have robbed the bank because I was cheating on my spouse at another location".
But the amendment is much broader than that. Since you don't know what other information the police are attempting to validate or even what crimes they're aware of, you have no way to know if "I was at $hotel with $other_person at $time" may incriminate you in this case or some other case.
However a judge could force you to testify in someone else's case. You can't broadly plead the 5th in that case unless the testimony is incriminating. Let's say you saw a murder happen at a hotel you were at while cheating with your wife. A judge is allowed to force you to answer the question.
The federal definition of misprision requires that, “(1) the principal committed and completed the felony alleged; (2) the defendant had knowledge of the fact; (3) the defendant failed to notify the authorities; and (4) the defendant took affirmative steps to conceal the crime of the principal.” See United States v. Baumgartner (6th Cir. Sept. 24, 2014)
(I only know because i am barred in maryland and dc, and virginia was the state that had adultery on the books)
It is unlikely to ever be held constitutional, so in practice, they use it as something to plea bargain to :)