"it's accepted without controversy in Japan."
While not completely true as an absolute, this is close enough to the truth to be a fair statement.
Even some 児童ポルノ wasn't illegal in Japan fairly recently. And like the other Japanese censorship laws, was brought in as much due to a desire to align with Western pressure and laws as due to any moral sense. (I'm not trying to diminish the second aspect here; I'm saying that both were strong pressures to create the law).
Remember that Japanese morals are much more Confucian than Christian. Breaking a law is immoral in Japan in and of itself because laws exist to create harmony and breaking harmony is immoral. So the fact that 児童ポルノ is illegal, and ロリコン is not forms a large part of the opinion of whether it is immoral or not.
Also remember that the Western hatred of pedophilia and child porn is partially Puritanism. We've successfully thrown off the Puritan yoke, but that part of our culture was deeply rooted, so rather than disappearing, some of it has transferred over to perversions that it's still acceptable to demonize, namely pedophilia.
And I think the Confucians have it right on this one. Pedophilia and child pornography is one of those things that needs to be banned and made illegal for social harmony reasons, and practicers of softer forms of it like ロリコン should be marginalized. Different from, but mostly compatible with the somewhat commonly held opinion that pedophiles are "sick, not necessarily evil".