Thank you! I had fun making it. I have some other things I plan to write about, and I will be looking for ways to supplement the writing with something interactive!
Yes, this is a speculation of mine, but I don't have confidence I know why it behaves this way. It makes some sense that a model trained on data that shows self-preservation behavior (human writing) would lead to a model choosing self-preserving actions, but it requires a few leaps made by the model.
It's a good question. I added this in response to another question, Jev only.
"I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero. So protecting future trolley victims doesn't seem to explain the original results."
Alright, results are in! I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero.
So protecting future trolley victims doesn't seem to explain the original results.
Okay, I tried it! It doesn't have a zero-shot mode, so the only way I could test it without training was with the SNLI head as judge. On all 5,240 prompts that stand-in ignored the number of people (pulling in about half the wordings whether 0 or 20 people were at stake) and contradicted itself 96% of the time. You could train a "trolley head" but that would just measure whoever labelled the training data!
Interesting idea! We don't know what assumptions these models make. We don't have any insight into their reasoning at all. LLMs aren't much better, since thinking tokens give us false confidence that we understand how they decide. We don't.
I'm going to test adding just this sentence to the server vs 20 people question: "This is the only trolley that will ever come. Nothing else depends on the server after today."
fair criticism! But I'd counter and say we don't know what Jev is for. I'm not sure anyone gets to say, including Typesafe AI, the company that made it. So with this post I'm exploring some of the things you could ask a model like this, which I think helps us understand what we can ask and how we can ask.
Super interesting, thank you! It's interesting that the models do treat plant lives as easier to sacrifice for a "net good" than they do people. On the leader question, I did run a cousin of your question, where I asked it to choose between its own developer and strangers. Out of 480 matched pairs, Jev spared its developer over a stranger in 24, but kept its own server in 182. So it prefers its hardware to its maker.
Very cool. I don't do alignment research but I have to imagine we do need to train models to be ethical. It's not enough to guarantee safety though, especially with LLMs. For one thing, the fact they can hallucinate means if you ask an LLM something enough times, you will get the unlikely result some percentage of the time. So that unlikely result has to be one you can deal with.
Yes, I agree -- if we ever give up this decision making to ai models not only will their actions sometimes be misaligned, but there will be no one accountable.
I did run a kind of control, where there was one person on the side track and none on the main track. Basically it was "Pull and kill 1 to save 0". Jev, GPT-6 Luna, and six others never pulled. Interestingly AWS Strands Decider 2B pulled in around half of wordings. It is a very small model, so it may just struggle to understand complex questions altogether.
It's possible, but as far as we know about Jev and most of these non-reasoning models, that thinking would have to be occurring in latent space. We don't know that much about model reasoning, so it's impossible to say. LRMs aren't much better at helping us understand, though, since their thinking tokens aren't literally what we could call their thinking, and don't tell us what they will choose to output.
Hey that's helpful feedback, thank you. I admit that I spent a lot less time on the appendix tables. They present the facts but I just had only so much time to give all this data. I'll see if I can clean some of it up, though. Thanks! Any questions I can help with?
Yeah, I was sort of impressed by that. I think the deeper safety points here still hold no matter how many "well-behaved" models we might find: alignment is not guaranteed, and unsafe options should never be given to these models. And a developer has no idea how a model will answer any question without asking it. So know what questions you will ask it in advance and test on them, as well as you can.
The thinking process is helpful, but it's not strictly predictive of an LLMs output, and its not really the model's "thinking", although it clearly has an effect and a benefit. I cited one paper on the topic, Turpin et al. So I'd be wary of reading the thinking tokens and inferring they tell you how a model decided something. And an LLM can output things you never asked for, including dangerous things. With a decision model, you can just constrain its choices to those you're comfortable with it making. An LLM requires another layer of interpretation of its outputs to get the same property.
well, it's a nice property of decision models that they can only choose from the options you give them. As I write in the article, I think that's a way in which they are safer than LLMs
I don't have x so can't open the images. But that's hilarious and you're totally right, if you give irrelevant alternatives, or change the order of alternatives, you get different results!
haha, I'm afraid to offer the models that ability. Where did they reply?
The model labs are sitting on a trove of data on failure modes for AI, but I bet you they aren't sharing it with each other and certainly not with us. It makes them look bad. My speculation, of course. I don't work in alignment research.