Some have likened AI's potential for change to technologies that made the industrial revolution possible. The industrial revolution was a great thing right? Everybody had more of everything after it was over. But what about during the revolution?
The truth is that a lot of people suffered a great decline in quality (and duration) of life during the industrial revolution until social change caught up with technological progress. e.g. In England, early in the revolution, it was still illegal for workers to change jobs. When the choice was stay at your current job or go to jail, it's not a surprise that wages stayed artificially low and worker safety conditions were appalling. The complete lack of environmental regulations and the resulting pollution is another famous example of how social change had to play catch-up with technology.
Right now, there is a lot of potential for AI to increase human suffering in the short term even if it is a net benefit in the long-term. At least some of the "AI ethicists" out there are trying to get ahead of the curve, and that's laudible.
You might want to read "Power and Progress - Our Thousand-Year Struggle Over Technology and Prosperity" by Daron Acemoglu and Simon Johnson. It's not really an AI focused book, but it will give you an appreciation of why some caution is warranted.