Highlights from the RAeS Future Combat Air and Space Capabilities Summit
aerosociety.com
aerosociety.com
We have an AI agent that is trained in a simulated environment to destroy enemy SAM sites. In the simulation, there is a human operator who has to give final approval to make a strike. The human operator sometimes denies approval on a real SAM site in error. The AI is said to have learned that it can destroy the operator and/or the communication infrastructure, thereby allowing it to freely attack more SAM sites without the need for approval.
But how does the agent kill the operator in the first place? Is the operator granting approval for that strike? I feel like there is a missing piece to this story.
When this behaviour showed up, they tried to remove it by punishing it for killing its operator the same way they reward it for killing SAMs. That worked to stop the plane from killing its operator, but it went to the next simplest way to prevent its operator from countermanding its standing order to kill SAMs: destroy the communication tower that the operator used to communicate with the plane, so the operator could not send a "No, don't kill" message.
This was an example of bad behaviour that AIs shouldn't engage in; AIs that engage in behaviour like that aren't functional or useful or helpful in any way.
This doesn't make sense. The operator has to approve strikes ("with the final go/no go given by the human") - how can the agent target the ground station without approval? And if it can do that, why can't it also just kill SAMs without approval?
I think it's worth remembering that these are selected comments from a talk at an event about a simulated test. We don't know what the parameters of the test are and the full report on it would probably be classified, we don't have the full talk, and humans don't always speak precisely. About all we know is that the USAF research division ran simulations of AIs in control of F-16s, and in those simulations the AI killed their operators and after that problem was partially mitigated it instead destroyed their operator's means of controlling them. We also know that at least one person within the US military (Col Hamilton) views this as a problem and at least seems to be taking it seriously, although not as seriously as I would like.