That's interesting.
I remember when I lived in Japan and was with my male friend visiting temples. I was 21 years old and I'm male. At one point he grabbed my hand to direct me to a temple and didn't immediately let go. I felt strange because in the US that would be misconstrued.
Are you sure Japanese people hate physical contact? I never really considered whether that's true or not.