As far as I can tell, the argument boils down to all that you wrote,
relative to what you gain. AFAIK a supersonic passenger plane doesn't shorten your total trip enough, compared to regular passenger planes, to create a new quality class of travel.
E.g. taking 5 hours to go to a place (2h flight + airport overhead on both sides) by a plane vs. 12+ hours by bus makes many destinations suddenly accessible to people to visit for a weekend. Except maybe on intercontinental, I don't see a supersonic passenger plane enabling another quality class of travel.