Note that the implementation did not provide any guidance about ethics in the system prompt, so the alarm is that the weights themselves may not ensure aligned behavior. I find this totally unsurprising, by maybe it's surprising to someone?
It is also as they say the first demonstration of this behaviour unprompted. They also show that this behaviour can happen without a "thought scratchpad".
It also seems like more capable = more likely to be misaligned and deceive.
I think it's a very interesting paper, surprising or not.
They are also arguably surprising elements.
It doesn't seem like RLHF is doing anything to mitigate this behaviour. If anything it seems to be the opposite.
I had always dismissed Asimov's "rules of robotics" as silly: nobody would ever design a mission-critical robot with indeterminate stochastic behavior! Maybe I should reconsider and re-read those stories, because people seem very eager to do just that.