Because it's a good 800 km from Tokyo, and 4 hours one-way even by bullet train. And because, A-bomb sites and Miyajima aside, there's not all that much of interest in the city of nearby.
On the other side of the coin, I also find it a bit odd to describe Hiroshima as a cliché. Quite the opposite, it's near-unique: Hiroshima and Nagasaki are the only cities ever attacked with nuclear bombs, and you can't get the experience of visiting them anywhere else.