Some other issues are that high speed rail is not green, and it is not affordable either.
So far as adding more passenger rail lines, it isn't easy. For instance, plans for HSR in California are unrealistic because you don't save any time from SF to LA unless you can go from downtown to downtown. Sure you can build track in the Central Valley, but it would be pretty hard to take enough land by eminent domain to build a track into the cities even if these people were not the most feared NIMBYs on the planet.