The risk is relying on AI to give a human information on whether or not to push the button and the AI gets it wrong but communicates so effectively the human trusts it without question.
An AI learning about and then executing a social engineering attack on a human (whether with malicious intent or not) seems like a reasonable thign to be worried about to me.