Actually, it's worse than it sounds. Travel time has a big impact on demand--but only within a small-ish window, about 2-4 hours. Outside of that window, demand does not respond that much to changes in travel time. That means that higher speeds are ultimately about trying to bring more destinations within the acceptable travel time window, but you also have to give up destinations within that window because the stop time penalties are too high.
Switzerland has the issue that it's just too small. With conventional 300-350km/h max-speed HSR, going from the eastern to the western edge of the country could be done in about 2 hours (recall that average speed is going to be less than max speed, particularly when stops are included), which is around the point where going faster means getting no practical benefit for people. It could serve international routes, but then Switzerland is bearing the costs without reaping the benefits, which the citizenry tends not to favor very highly.