It doesn't matter that adultery isn't a crime: you're being questioned by the police, and any answer has the possibility of including details which might seem incriminating for whatever they're actually investigating, regardless of your innocence.

It seems like you're approaching the 5th amendment as a direct point/counterpoint arrangement: the police accuse you of robbing a bank, you know you didn't because you were cheating on your spouse, you can't take the 5th because cheating on your spouse isn't a crime, so you must answer "I couldn't have robbed the bank because I was cheating on my spouse at another location".

But the amendment is much broader than that. Since you don't know what other information the police are attempting to validate or even what crimes they're aware of, you have no way to know if "I was at $hotel with $other_person at $time" may incriminate you in this case or some other case.