http://en.wikipedia.org/wiki/Prostitution_in_Japan
As you say, it takes a lot for the police to get involved - I'll speculate that they only pay attention when the monthly envelope of cash is light.
That isn't the model pushed by most advocates of legal prostitution that I've heard - most want prostitution to be legal in the same way that restaurants are. It actually takes very little to get the authorities to come take a look at your restaurant - complaints of a roach or the computer randomly selecting you for an inspection are usually sufficient.