My theory is that the cultural inclination to sexualize very young girls in Japan has created a great number of adult men who are unable to be sexually attracted to adult women.

It's a cultural disorder, not a problem of any individual man being "defective".