Do you mean legal residents, or not showing their ID is legal? Because if police ask for a foreigner's resident card, they have to present it. [1, Q41]
[1] http://www.immi-moj.go.jp/newimmiact_1/en/q-and-a_page2.html
If yes, they must provide the reason. [1] At that point, you may have to show them your card depending on the circumstances. They may leave you alone, however.
My experience: I was once taking photos in a dark alley for some cyberpunk type shots around 9 p.m. I was approached by two officers who asked, "What are you doing here?" Kind of a trick question, because then they said, "Oh, you speak Japanese. Do you have your resident's card?"
I complied here because of the optics. After complying to show my card, my bag and me were fully searched which I was not okay with. After that I experience, I became well acquainted with my rights because, at that time, I did know them well.
1 - https://www.japantimes.co.jp/community/2014/04/02/issues/rig...
[1]'s source / reference: http://www.debito.org/shokumushitsumon.html
Police have almost unlimited power in Japan, and gaijin have no rights at all. Japanese citizens have some, but in general will not exercise them due to their cultural upbringing.