If you go through Roppongi you'll see them, largely standing around. And in most shopping streets you'll see them patrolling on bicycles or in cars. They don't tend to stand around at malls or train stations, agreed, is that normal where you come from? (It wasn't the norm when I lived in the UK).
In the US it's common to see police out in areas like that. I looked it up and it seems NYC and Tokyo have the same number of cops.