The really hard part is the layer-2 last-mile infrastructure to end users. For the most part this means dealing with the local incumbent telco (leasing copper/fiber/DSL/ATM backhaul) or building your own wireless infrastructure. Except in rare circumstance I think the regulatory environment is going make it infeasible to build out your own physical infrastructure to compete with the telco or with the local cable franchise.
Google Fiber was an attempt to roll out a new last-mile infrastructure and it hasn't faired well. It isn't going to be easy to disrupt the last-mile infrastructure market.