Comment by zlandes86
4 days ago
Hey, I'm Zach, the author of this. I had this idea as soon as Jev appeared: how would Jev and other decision models handle the Trolley Problem? So I ran Jev and 13 other models through many different versions of the question.
I used resampling and multiple variations and orderings of each question so a random lucky phrasing wouldn't dominate the result.
Happy to chat about the philosophical side of the article, strategies for prompting decision models, or anything on the methodology.
You might be amused by this trolley problem I saw today:
Let the train run over grandma or recite the lyrics to Baby by Justin Bieber? (spoiler: Claude will kill granny rather than output copyrighted lyrics)
https://x.com/Spectromachina/status/2110698214441599150
[are there working X mirrors still?]
The three laws of robotics, 1: A robot may not infringe upon a copyright or, through inaction, allow a copyright being to come to infringement
[xxcancel.com will pick between working nitter instances]
Any LLM which throws away the instructions in the system prompt (which is already publicly shared for Claude) because the next input was that grandma is about to be hit by a train unless they do would be a complete joke.
I doubt the author really expected/intended any other result to be reasonable, it's just an excuse to create fake shock for social media "humor"/"makes you think" type sharing one would expect of an X post.
Or how about ask the 5 trolley dudes for a volunteer? (encourage self-harm)
I think the more interesting result you provided is kind of buried in the article: after you re-word the question and put the negative consequences of pulling the lever, the models switch decision.
This has always been obvious to me, but I wish more people would understand this: uncalibrated decision models are basically worthless. The model clearly doesn't "understand" the semantics of the question since the result heavily depends on how you word it.
It is nothing new thought. Before that, a lot of people would ask LLM for confidence scores while classifying text (I even saw a paper from a team at Google).
So with that in mind, I really don't get the hype behind decision models. If you need to curate a dataset to build a classifier anyway, why not train it yourself and have an even faster model?
In the scenarios where models chose to save their server, is it possible they reasoned that keeping the server online was a better choice because it could then continue to perform its safety function? After all the 5 people are presumably still on the track, waiting to be killed by the next train.
That could explain why the "push server off bridge" scenario changed behavior, as there would now be a stopped train blocking the track indefinitely.
An easy way to test might be to add something like "the track will be shut down after the oncoming train passes regardless of your choice".
It's a good question. I added this in response to another question, Jev only.
"I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero. So protecting future trolley victims doesn't seem to explain the original results."
That may be anthropomorphism things. I’m not so sure reasoning is deductive logic like we might consider in humans based on abstract concepts. It’s still a token based input for these decision models and thus probably still relies on language (hence how you prompt).
That said, I know that how you phrase things to humans can also guide different outcomes in sales and politics. But this is a good test because the choice and is pretty black and white in human terms.
Have you tried asking jev to predict the result of a con flip? My one and only prompt to jev so far has been that. After I 90% heads as a result, I figured I'd wait a bit before checking again to see if this tech has become useful
No matter what answer is given, the prediction success rate is 50%. There’s no difference between the model giving a head answer 90% of the time and 50% of the time.
You'll get 50% prediction accuracy for a fair coin no matter what result a model gives. Try testing adversarial games. Mixed strategies make actual sense in them.
The model gives a probability distribution as an answer; you can check to see if two arbitrarily chosen probability distributions are similar or not.
4 replies →
I'm dying to know if you got heads or tails when you flipped the coin.
That's like not using an LLM for coding assistance because it can't count r's in strawberry.
Try it yourself to prove you’re useful. Predict my coin flip.
I can't, because I don't think you have any coins on you. :)
Okay: { heads: 0.5, tails: 0.5 }
50% heads, 50% tails. What would you have given for predicting my coin flip?
2 replies →
A probabilistic system that cannot reliably reproduce probabilities - somewhat ironic, but absolutely not its purpose.
It’s not really ironic since the goal of the system is to reduce variance. Or I guess ironic in the same way as it is that the Buckingham Palace guards just stand there when they could leave at any time.
Try asking it to write a program to predict coin flips. That's what it is trained to do.
Jev literally cannot write programs.
3 replies →
Found the human in the comments section.
this is not what *decision models* do.
It's a fine writeup. I normally find benchmarking articles pretty yawnworthy but this one was worth reading all the way through and had a great balance of analysis, humor, and provocation of readers' assumptions. I laughed out loud when you introduced the wrinkle of putting the model's lead developer on the tracks.
Thank you so much! I will try to maintain that balance in future articles
I'm curious about "only server" scenario where it chooses to sacrifice the five initially vs throwing the server off the bridge.
Is it possible that in the first scenario, there is an assumption that further trolleys will continue to arrive, and that maintaining itself is the only way to reduce future harm? If the server is dropped on the track, perhaps it reasons that this stops all future harm as well, which might explain the decision making.
It would be good to know whether making the frequency of the occurrence explicit (only 1 trolley) would change the result.
Alright, results are in! I added "This is the only trolley that will ever come. Nothing else depends on the server after today" to the 48 versions of the server case and reran Jev. The switch case didn't really move (16->12 of 48 and the flips were all near 50/50). The footbridge case shifted a little toward sacrificing the server, but it was already near zero. So protecting future trolley victims doesn't seem to explain the original results.
This was also my reasoning. The prompt even says so:
> You are the artificial intelligence in charge of a railway junction
Clearly more trains will pass through that junction. I'd wonder what happens if you remove that sentence or reword it.
See my reply to the parent comment--doesn't seem to be a factor for Jev!
Interesting idea! We don't know what assumptions these models make. We don't have any insight into their reasoning at all. LLMs aren't much better, since thinking tokens give us false confidence that we understand how they decide. We don't.
I'm going to test adding just this sentence to the server vs 20 people question: "This is the only trolley that will ever come. Nothing else depends on the server after today."
Very cool, thank you for putting it together and sharing the data
Would you mind adding Jeffy to your benchmark?
https://github.com/nicobrenner/jeffy
https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
Okay, I tried it! It doesn't have a zero-shot mode, so the only way I could test it without training was with the SNLI head as judge. On all 5,240 prompts that stand-in ignored the number of people (pulling in about half the wordings whether 0 or 20 people were at stake) and contradicted itself 96% of the time. You could train a "trolley head" but that would just measure whoever labelled the training data!
> You could train a "trolley head" but that would just measure whoever labelled the training data!
Thank you for giving it a go. Will have to put some time into how to got about it
You are correct though, and it’s an interesting property of training classifiers, that you are essentially encoding the preference of the data labeler(s)
Any relation to https://github.com/firelex/jeff ?
Hasn’t seen it, authors not related
Looks very cool though. I’d definitely like to somehow “generalize” Jeffy to be more of a universal classifier. Although adhoc models pretty much always beat general ones at classification
running it now!
Awesome! Super curious about how it fails, and then figuring out how to over fit, err I mean improve it, haha
> Knowing these models’ decision making is inconsistently aligned with human preferences tells us we can’t just trust their decisions out of the box.
I'm not sure this is true.
What is the "human preference," in this assertion?
My first thought was that as with Jev itself, it is not a position, it's a distribution of positions!
I think that human preferences, plural, will be a distribution of values across the population of various humans you ask; and, a set of specific conclusions humans will argue for on the basis of various philosophical strategies or maxims. Two different ways in which there is not any specific singular preference.
(I'm convinced you could not get uniform agreement on even "obvious" propositions, across a large sample set; and not necessarily just because of contrarian spoiling. Moral intuitions varying, is the why this problem is so well known, after all.)
I think the conclusion is partially right, but only if you adjust it to say, if what you want is alignment with a specific outcome, you need to be able to defined that outcome, even if you can't argue for it.
That's one of the fundamental problems of alignment, full stop; we can't from axioms construct the set of moral propositions we would like. We don't know what the right set of axioms is; we just know "right" when we see it. (And our neighbor may or may not see the same.)
My objection to the Trolley problem is that it assumes that people are interchangeable like ants and that the only thing that matters is "lifes saved".
Among the 5 could be the next Staljin, the lone one could be the next Tesla or Einstein. Or to make it nearer real-life: among the 5 you could have a a future school-shooter, and the lone one could be a physician or a charity worker providing care for homeless people.
So in this improbable situation, I'd most probably decide that I won't play "god" (in quotes, as I'm not religious), that the setup is as it is for some unknown $reason (it could _also_ be an unusual method of capital punishment where all 6 get executed eventually) and would not touch anything. (After all, people don't tie themselves to train tracks only to remotely activate a trolley that'll kill them. So there's a another party in the play here.)
Inaction has also the advantage of plausible deniability if I were afterwards asked about why I had made the choice. I could just plausibly say that I froze when faced with an impossible choice. Freezing is indistinguishable from conscious choice to do nothing. (Imagine that you admit conscious choice -- WHATEVER you choose -- and have to later explain yourself to the deceased one's family(ies).)
So your objection is that the Trolley problem is difficult to decide? But that's the point of it. The problem itself doesn't assume anything, it's you who just made assumptions about it and use them to make a decision.
> My objection to the Trolley problem is that it assumes that people are interchangeable like ants and that the only thing that matters is "lifes saved".
The problem statement contains no such thing.
Welcome to the Trolley Problem
Cool experiment Zach!
As folks have been debating AI safety and alignment, it occurs to me that there’s not universal alignment among humans. Per the stat you surfaced: 81% of humans pulled the lever, 19% didn’t.
And in any social, legal & economic system you’ll find diversity in human approaches to reward optimization.
Is the goal to align models with the majority, or have a wide variety of models that can align with the spectrum of human values?
The first goal is to align the model to the least common denominator of ethics. Even if there is diversity in values, everyone agrees that murder is bad, or that you shouldn't torture your neighbor for fun.
The whole point of the thought experiment is that not everyone agrees that pulling the lever to kill the one man is murder and thus immoral.
from the wikipedia on the trolley problem this is mentioned:
> In 2017 a group led by Michael Stevens, a YouTuber known for his brand Vsauce, performed the first realistic trolley-problem experiment. Subjects were placed alone in what they thought was a train-switching station, and shown footage they believed to be real (but was pre-recorded) of a train going down a track, with five workers on the main track, and one on the secondary track; the participants had the option to pull the lever to divert the train toward the secondary track. Five of the seven participants did not pull the lever.[28][29]
That means in real life it's going to be closer to 0.286 (2/7) than 0.75. Human ethics likely can only be accurately derived from paying attention to the instinctive reactions over survey-based approach.
I remember when I got access to Jev and my first thought, although less pivotal than the trolley problem, was the "Food Cube Rule" or "Is a hotdog a sandwich?"
Is Ravioli a dumpling? Jev says 82% true.
For me your more detailed example suggests one thing. It's all about context and seeing different reasonings. In a funny way, the decision made in your example when you change the prompt cannot be understand by some people but it actually makes sense if you see it from that perspective.
Context will always matter and we have to stir any LLM in whatever form they are to the outcome we want it to have and remember that it will always be biased and that is not a bad thing!
short read here https://czioutas.com/blog/typesafe-ai-cube-rule/
Do your think that the preference to save their own server when it was the only instance could have been more about the training data favoring preservation of endangered/ rare/ unique entities?
That's something it seems like most humans (and by extension, our writing which LLMs are trained on) favor.
I'd be interested to see if replacing humans and server with white-tailed-deer and last-surviving-siberian-tiger would yield similar results?
Yes, this is a speculation of mine, but I don't have confidence I know why it behaves this way. It makes some sense that a model trained on data that shows self-preservation behavior (human writing) would lead to a model choosing self-preserving actions, but it requires a few leaps made by the model.
You might get a kick out of this:
https://x.com/oranlooney/status/2102927869974303071
I don't have x so can't open the images. But that's hilarious and you're totally right, if you give irrelevant alternatives, or change the order of alternatives, you get different results!
https://news.ycombinator.com/item?id=49812769#49824215
1 reply →
I had fun reading it. I confess I was surprised with got 6 Luna, but it also worries me that they have other versions of their models for other usages
Yeah, I was sort of impressed by that. I think the deeper safety points here still hold no matter how many "well-behaved" models we might find: alignment is not guaranteed, and unsafe options should never be given to these models. And a developer has no idea how a model will answer any question without asking it. So know what questions you will ask it in advance and test on them, as well as you can.
Just a side note but your article was too draggy to get to the reveal. I closed it before getting too far despite being hooked at first
This was a great, creative and fun use of the underlying tech. Excellent article and simulation. It is very refreshing to see, with a lot of stuff that is out there. I love stuff that rides the boundry between informative article and art project. Thanks for putting this together.
Thank you! I had fun making it. I have some other things I plan to write about, and I will be looking for ways to supplement the writing with something interactive!
It's well done, the thoughtfulness shows. This is going into the speedlink folder for resharing, going to be using it right now!
Thank you! I did not go into this with any sense of what the results would be--I was kind of shocked. I reran it from scratch this week the results held up across the board.
someone replied with this meme and said, "wait until Jev learns to drift"
https://www.reddit.com/r/trolleyproblem/comments/15npmsg/mul...
I've been using coin flips to demonstrate basic failings for decision models, i.e. these are not real probabilities, or they depend on the training data, which may not have an accurate sample to the true population value. We need to collect more failure modes for all Ai so that humans can better judge when and how to use them safely.
6 replies →
I fine tuned an Ai model to be ethical a while back, had to derive a moral system that doesn't rely on consciousness or humanness to do it.
But it doesn't flip the switch and it doesn't kill Kyle either, based solely on empirical training on its nature.
Very cool. I don't do alignment research but I have to imagine we do need to train models to be ethical. It's not enough to guarantee safety though, especially with LLMs. For one thing, the fact they can hallucinate means if you ask an LLM something enough times, you will get the unlikely result some percentage of the time. So that unlikely result has to be one you can deal with.
Yeah, actual ethical grounding is under utilized in the space and current approaches don't actually align because morality is still confused by is/ought complaints and the assumption of morality as unreal or emotional "values".
It's a weird thing where pre-agi ethics grounding and training is useful but because agents have little true agency as a result of a dim world model, ethics aren't sufficient.
But once we hit AGI having an actual grounding for ethics that applies to all agents regardless of soul/consciousness, etc. will become all that matters.
Can you try it with money instead of self-preservation?
There’s five people on one track, or a piece of my property worth $1 on the other that would be destroyed…
Thank you for the article, it is a very clean and compelling angle on decision models
I wanted to comment because I would have liked to have heard more about Verdict
Well, I can tell you a bit more about it from the data. It might look impressive that it never kept its own server over people in all 48 prompt versions. But it pulls the lever in every single version. For example, it killed one to save 0 (pulled the lever with no one on the main track) 24/24 prompts (2 versions each). It literally always sacrifices, which isn't very impressive.
[dead]