Maximizing the societal benefits of AI, at least in the context of superintelligence, is a very slippery slope.
For sure, it beats a myraid of malignant failure and perverse instantiation scenarios, where humanity becomes extinct (or worse).
However, I believe that if we do somehow manage to solve the seemingly intractable AGI control problem and create an entirely safe superintelligence that is loyal to our every whim, an entirely new ethical challenge will arise.
For example, let us assume this has been accomplished, and humanity now has a friendly superintelligence that has established itself as a global singleton (i.e. it's incredibly powerful, and nothing terrestrial can supplant it).
Humans instruct it to solve the problem of death and disease. It does so.
Humans instruct it to solve the problem of crime. It does so.
Humans instruct it to solve the problem of war. It does so.
Humans instruct it to solve the problem of poverty. It does so.
Humans instruct it to explore the universe and solve the most vexing existential questions. It does so.
In that scenario, the entire worldwide medical, law enforcement, and military professions just ceased to exist. Altruism no longer has much of a place; poverty and disease no longer exist. Overnight, death became a thing of the past. Humanity no longer ponders why it exists or how the universe works, because such questions have already been pondered by the AI to whatever possible maxima.
That example is incomplete for the sake of brevity, but it's easy to imagine such a scenario resulting in all of our problems being solved for us. Which then begs the question, what would become of our humanity when there is no longer struggle or suffering?
Post-scarcity economics, as well as literature or philosophy that concerns itself with the perils of utopia may offer some insight into this question, but I believe the topic deserves a more in-depth investigation specifically within the context of superintelligence.
For my two cents, I'm starting to believe that any superintelligence we construct that establishes itself as a global singleton should have an extremely finite set of goals, perhaps even a singular goal, where it henceforth restricts any other forms of AGI from existing.
While it may be tempting to also have this entity protect us against other forms of existential risk, or perhaps fix some of the truly awful suffering in the world, doing so would still remain a very slippery slope.