I don't think geography is the explanation. US geography is great for passenger rail. Japanese geography is terrible for it (too many mountains). In my opinion Japan's passenger rail situation today would be no different than the US if American rail companies had managed Japanese rail lines throughout the 20th century.
It's nice to imagine that geography is the reason, and so everything is as it should be, this was the fated result. But that's simply not the case.
Of course, rail is not a good fit for a weekend getaway from LA to New York. Even in Japan, where you can ride the bullet train all the way from Kyushu to Hokkaido, almost anyone would book a flight for that unless they really hated flying. But that doesn't mean that rail isn't well suited to American geography.