Typically (not always), if someone goes to prison, it is because of them using up many chances to avoid going -- or they did something particularly heinous. Japanese prisons are not flooded with minor drug violators like US prisons are.
First, the police I've met in Japan are VERY willing to look the other way rather than convict someone on a serious-but-relatively-minor offense (e.g., drinking and driving while not "drunk"). I have also known them to handle delicate situations in very diplomatic ways (e.g., appropriate but definitely illegal street justice).
The stories I have heard of people going to jail involved that person breaking the law repeatedly and with multiple warnings (usu. with accompanying letters they had to write saying that they would not continue to behave in a criminal way).
In the more provincial areas (even the small cities), the police actually have a pretty good feel of what is going on in their towns -- the speed of gossip is impressive. If anyone is acting up, it would not be surprising for the police to find out and just go have a friendly talk with that person reminding them that doing $CRIME is, well, a crime that might have very unfortunate consequences. That discussion usually puts a lid on it.
I will also add that for a very large portion of the population, the peer pressure to conform goes extremely far in discouraging people from doing things that would put them in prison. The idea of prison itself is just a foreign idea for most people.
I haven not recently updated my reading on the Japanese criminal justice system (most of my knowledge is 10+ years old), but I don't recall reading much that made me feel like there were many prisoners were in prison unreasonably. A few were, but just a few, and completely different than the US where it's obvious that quite a few people are in prison for things like minor offenses, getting framed, not being able to afford decent representation, etc.