I wonder if the three laws of robotics are already weaved into the LLM. Seems like a necessary step for this kind of usage.
Where they argue that basically having an AI follow these laws is impossible because it would require rigorous definition of terms that are universally ambiguous and solving ethics.
Remember what happened in Isaac Asimov's iRobot?