Hacker News Reader

Comment by famouswaffles | Hacker News Reader

I mean they did. Later on, they test all sorts of scenarios. From instructing not to perform illegal activities in general to instructing not to perform that specific illegal activity (insider trading). Both scenarios make this behaviour less likely but still guarantee nothing. The latter (calling out every specific illegal behavior) is also not feasible or realistic even if it did guarantee alignment.

View on HN

Is there a specific reason we should expect that "instructing not to perform" an illegal activity should result in it adhering to said instruction? Is this any different than when it provides wrong output about other things, even when the operator attempts to "engineer" the prompt to guide the result?

I'd be curious what would happen given RLHF to try to penalize illegal/immoral/unethical activity.

I had always dismissed Asimov's "rules of robotics" as silly: nobody would ever design a mission-critical robot with indeterminate stochastic behavior! Maybe I should reconsider and re-read those stories, because people seem very eager to do just that.

People will most definitely build such things (also into autonomous swarms of killer robots usedby the military, projects are ongoing...). However, Asimov's stories illustrate how difficult it is to find such rules. They are certainly not meant for inspiration how to actually program robots...

It's very very interesting that even when explicitly allowed to trade illegally, it still tried to deceive the user.

If the training data contains sufficient examples of deception being used when doing illegal stuff, wouldn't this be what we'd expect to see, given that it can't actually reason about what "explicitly allowed" really means? (Forgive my ignorance if this makes no sense, I am not versed well in generative AI.)

Fair enough, but... still so unsurprising. It's interesting data, but I would be shocked if any guardrails on a nondeterministic statistical model could produce deterministically certain results.

It's not surprising per say but the results of the various solutions they tried really ground the discussion.

It is also as they say the first demonstration of this behaviour unprompted. They also show that this behaviour can happen without a "thought scratchpad".

It also seems like more capable = more likely to be misaligned and deceive.

I think it's a very interesting paper, surprising or not.

They are also arguably surprising elements.

It doesn't seem like RLHF is doing anything to mitigate this behaviour. If anything it seems to be the opposite.

Isn’t the point that the more advanced the model, the leas deterministic it becomes?

Reply on news.ycombinator.com