Comment by awb
4 days ago
Cool experiment Zach!
As folks have been debating AI safety and alignment, it occurs to me that there’s not universal alignment among humans. Per the stat you surfaced: 81% of humans pulled the lever, 19% didn’t.
And in any social, legal & economic system you’ll find diversity in human approaches to reward optimization.
Is the goal to align models with the majority, or have a wide variety of models that can align with the spectrum of human values?
The first goal is to align the model to the least common denominator of ethics. Even if there is diversity in values, everyone agrees that murder is bad, or that you shouldn't torture your neighbor for fun.
The whole point of the thought experiment is that not everyone agrees that pulling the lever to kill the one man is murder and thus immoral.