But to your point, more than half the time I take that trip there's some delay on the rails, usually in the CT corridor.
Or the time that a woman managed to lock herself in the bathroom without anyone else knowing. And there were no lights on in there. And it was night. So she spent a few hours screaming for help in a stinky bus bathroom, with no help to be found.
Amtrak's various unfunded fantasies for upgrading the corridor have usually included a dedicated pair of tracks for Amtrak for a reason. (I say fantasy because the proposals are always in the $115B+ range and every time I blink the number is probably going up, and that is an order of magnitude more money than Amtrak has ever had in its life.)
To put this in context, by the time suburban development started in earnest in the US, the private railroads were in free-fall after the war. It took until the '70s for Amtrak to put the (passenger) divisions of said railroads out of their misery, and Nixon did not anticipate Amtrak surviving the decade; Amtrak was supposed to decline out of existence. It never quite did that, but it also never quite got back on its feet, and so now the passenger rails are quite lacking.
Is that not the case in the Northeast Corridor?
The particular problem segment being discussed here does have four tracks, but it has commuter, regional, and intercity services. This is causing congestion problems; IIRC UK HS2 is being built because of a similar problem where they need to get the intercity trains out of the way of fast regional services.
With the amount of padding in the schedule required to run this effectively, it's possible that running an all stops service might actually be faster because it wouldn't require as much coordination.