First, IANAL, and I am not an expert for the situation in the US. So you're probably right.

But in the UK, you can go to jail for not disclosing your cryptographic keys. https://en.wikipedia.org/wiki/Key_disclosure_law#United_King...

Here is a story of a person who went to jail for not disclosing his keys: http://www.theregister.co.uk/2009/11/24/ripa_jfl/?page=1 He said he kept silent "because of principle", but I guess he might also have gone to jail if he told them "I don't remember" - Officers would not believe him anyway.