I too have always found this argument to be a non sequitur, for the exact reasons that you identify. If you attribute the emissions and pollution of the entire fossil fuel economy to the people who refine the products that go into the fossil fuel economy, there is nothing that can be accomplished with the analysis.

On the other hand, if we are serious about global warming and pollution, we need nuclear energy immediately (including the safe nuclear fission technologies which we've had for a few decades now, as well as continued research into fusion).