Why should a AGI save billions of lives?
I don't know which of various meanings you ascribe to "AGI", but the most general sense has no implications beyond the initials: is it artificial, is it "intelligent" (and that in the Turing or Dijkstra senses of what can it do not is it really thinking), and is that intelligence "general purpose".
So I expect AGI to do stuff we might not understand but further in the future we will look back with enough hindsight and we will understand why it did what it did and we will agree that it was the best action to take at that time. Just like children lacking understanding of why their parents tell them to eat bland veggies or do stuff they don't want to do even though it is good for them in the long-term. I expect this to be the main source of conflicts with AGI in the future.
• https://en.wikipedia.org/wiki/Principal–agent_problem
• https://en.wikipedia.org/wiki/Friendly_artificial_intelligen...
There will always be someone who asks "How to kill billions"
This is one of the reasons I, overall, agree with the OpenAI position that models should be kept secret until proven safe even though there are examples of safety researchers only being able to perform some of their safety research due to open models.
It's like… OK, so regardless what you think of it, you've probably heard some people blame COVID on a lab leak: I'd say that giving out increasingly advanced models for everyone to play with at home is like giving out increasingly advanced genomes in a world where everyone needs to have their own home gene synthesiser, including highschool kids for their homework — it's absolutely fine until that gene synthesiser reaches a certain level, and then suddenly everyone catches an airborne version of HIV.
Labs may still leak (IIRC, wasn't GPT-2 leaked?), but that's how I think we should be treating models: dangerous until proven safe, not safe until proven dangerous.
"""
I also want our future AI to be literally incapable of wanting to harm humanity, akin to in software "Make invalid states unrepresentable". But, an AI which cannot distinguish good from evil is definitionally Machiavellian; and if a model can distinguish them, and if the model is distributed, then it's almost trivial to find and flip that specific axis. But I did say "and if the model is distributed": if it isn't distributed, then the "switch" can be kept out of sight and away from fiddling hands.
There is another possibile safety mechanism, which is at least worth considering: if all possible moral issues are squashed down into a single dimension (something I've previously noticed humans did, but only by going past it) then anyone with a copy of distributed model will only have one single good-evil axis to modify — they would still be able to make a demonically evil moustache-twirling villan, but they wouldn't be able to accidentally destroy the world by turning off all the dials except the one saying "maximise shareholder value by maximising the number of paperclips", nor would they be able to run a dictatorship by saying "I am the only person who matters, everyone else is my slave".
The downside, and yes there is a downside, is that this will be the morality of whoever makes the model, and basically everyone disagrees on what "good" is. For example, even limiting the world to those who follow the Ten Commandments, "thou shalt not kill" is often regarded as so incredibly obvious that it shouldn't need to be stated, and yet that specifically has sometimes included and sometimes excluded each of: meat, abortion, war, the death penalty, and suicide — can any of us truly imagine how much broader the range and depth of disagreement is in agregate, or is this, too, an unthinkable thought?
""" - https://benwheatley.github.io/blog/2024/04/30-13.54.02.html