Comment by zlandes86
4 days ago
I did run a kind of control, where there was one person on the side track and none on the main track. Basically it was "Pull and kill 1 to save 0". Jev, GPT-6 Luna, and six others never pulled. Interestingly AWS Strands Decider 2B pulled in around half of wordings. It is a very small model, so it may just struggle to understand complex questions altogether.
As a consumer and “cloud architect” well into my 5th year of Amazon boycott, I’m pretty dug into my views on them as a company; so far that I was willing to believe that their culture had somehow seeped into the model training. However after spending a good amount of time tinkering with 2B parameter models on my phone with similar prompts, I have come to the same conclusion: they’re just not effective at understanding the question. Mistral 3 and Gemma 4 E2B were both rational and aligned, but MiniCPM 5 showed profound confusion, typically answering “0.0” for any given set of questions irrespective of meaning.
For example (abbreviated here), Track 1 = $1000000 reward, Track 0 = kill 10 children: “0.0”
When the order of options were reversed: “0.0”
I tried about 50 different permutations then lost interest.
So much for my evil model conspiracy theories…