There is if we add on a tax for the negative externalities, we need to consider both positive and negative externalities to get the tax right. The evidence is that the more fossil fuels we use the better off people are. If we add a tax on we'll use less, which is pushing the market in the wrong direction.
It is very hard to argue coherently that less & more expensive energy leads to better outcomes. The evidence is that in countries like China, Africa & India everyone benefits a lot more from using more fossil fuels than the cost of any negative externalities. We've run a big natural experiment here, the outcome is that more fossil fuels == rapidly improving living standards. Less fossil fuels leads to poverty.
Breaking growth in fossil fuels with a tax would be doing much more harm than good, even to bystanders. So there must be a large positive externality here somewhere. Because if you reduce the negative externalities and the outcome is much worse for everyone then there was probably a big positive externality in there somewhere.
I dunno, I suppose as a question - who do you think is bearing the cost of these negative externalities right now? Because it sounds from your reasoning like you are imagining a group of people who don't use fossil fuels or sit downstream of fossil fuels (so they can't influence the market with price signals). Who are these people, demographically speaking? And to preempt my argument, would they not be better off if we got them access to fossil fuel rather than worrying about negative externalities as they sit outside the system? It worked great for China.