I totally agree with what you've said, although I personally think that in terms of long-term misery that fossil fuels have the potential to be just as damaging as slavery.
To me that says that you are either drastically overestimating the potential misery due to fossil fuels or drastically underestimating the actual misery that was caused by slavery.
Perhaps you could start by answering this question: do you think I am drastically underestimating the potential harm from fossil fuels, or drastically overestimating the actual harm that was done by slavery? That will help me to "calibrate" where you are coming from. (And if it's the first of the two, as I suspect, can you give some specifics about why?)