For me the bigger negative externality is in foreign policy. I don't want any part of overseas adventures for oil. The financial costs alone are astronomical.
At the same time, I do benefit from transportation infrastructure in terms of shipping and purchasing goods which are transported. But if it were my decision I'd say just privatize the whole thing.
If a private road pollutes or otherwise harms anyone, then they should be liable. Problem solved. Lawyers have every incentive to claim damages.
About the abject terror, just build your way up. You can do it. Once you try it you'll see that it truly is as easy as riding a bike.
Back to your main point - it is not selfish to not want to ban petrol/diesel vehicles. I am ready to share the road with anyone.