You're forgetting the immense networks of rural roads. With high speed trains you can't just throw a level crossing up, and wait for a pickup full of chickens to get t-boned at 220mph. And then there is the wildlife - you'd need high fences the entire distance to stop wildlife and stock from wandering onto the tracks, but that also causes other issues with wildlife corridors.
Those rural areas are also largely divided into private property holdings, which have to be resumed meaning compensation for all the property owners.
You're right, the problem is political, in that the politics of the country is set up in such a way that a centralised body cannot just go and throw railways everywhere. But that is a design feature, not a bug. The same governmental processes that stop people ending up with a nuclear waste dump next to their town also apply to railway lines. You can't just plan this stuff from a central office in Washington and make it so.