Not sure why your comment is downvoted, because it hits the issue dead center, namely it's completely possible to pass either of the required tests (N2/J2) and not being able to speak a single word of Japanese in a live conversation.

That's why at least one category of applicants abusing the visa (Chinese) will continue to do so without any issues.