Yes, that's the price of a government-granted monopoly.
> If so, where do we draw the line?
If you don't have a government-granted monopoly, you don't have to be neutral. For example, that's how the Internet backbone works.
Yes doing the trenching and tunneling through various neighborhoods is disruptive and requires heavy machinery, etc. but that's just to create a hole in the ground. Once some conduit is laid (as a public good by the municipality?) it should be relatively easy for new carriers to come in and pull fiber or what-have-you. You just have to make the channel big enough for e.g., several cables and then you lease the space to the x highest bidders for five years at a time where x > 1.
There are probably better ways to finance things and minimize the disruption, etc. but overall it doesn't seem infeasible to me.