The root of the problem is the legal difference between POTS (Plain Old Telephone Service), which has been around a long time, and broadband, which came much later and had the POTS environment as an example of what to avoid (from the POV of Comcast and their ilk).

The old dialup ISPs were allowed to resell internet service on the local provider's POTS lines. There was quite a lot of competition, and the service was generally excellent, even generally better than the carrier's.

There is no requirement for broadband providers like Comcast to allow resellers. The barrier to entry for laying other broadband lines is huge; Google is one of the few who can do it. So unless there's a quantum leap in wireless to the curb, there will be no meaningful competition to amoral corporations like Comcast; we're stuck and it will continue to suck.

The real root of the problem is lack of regulation. A scarce resource like the last mile copper and cable, is owned by the oligopoly of Comcast, AT&T and a few others. This constitutes in essence a monopoly, which are illegal for obvious reasons and led to the breakup of the old AT&T. Other countries have a last-mile sharing requirement, like for example in Germany. This provides at least a modicum of competition and consumer choice in the ISP market.

[EDIT] Corrected wrong assumption that old AT&T was government owned.

Unfortunately I think this is the experience of most people in the U.S. At least in my adult life, the choice has mostly been between AT&T and Comcast. I was lucky to get 100Mbs for $50/mo for a while (obviously with a small ISP), but that was just a particular building that happened to have a point-to-point wireless setup on the roof. Now things are worse... I happen to live somewhere that actually has fiber in the ground (thanks to some federal grant the city received) but there is no service provider that uses it! I would be happy to ditch AT&T/Comcast for life if it were possible.

The root problem is that telecommunications is similar to utilities in that it is a natural monopoly.

Basically imagine if someone owned the only bridge into San Francisco. They could charge whatever people were willing to pay. But you would think if they charge too much money or scream obscenities at everyone who drives through, then someone could build another bridge and steal their customers. The problem is that if anyone builds another bridge the bridge owner could stop screaming obscenities and lower his prices. Then the second bridge could not make the return on capital. The prospective business owner and current owner know this so the status quo of one expensive and crappy bridge remains.

[0]https://en.wikipedia.org/wiki/Natural_monopoly