Comment by aaroninsf

> Knowing these models’ decision making is inconsistently aligned with human preferences tells us we can’t just trust their decisions out of the box.

I'm not sure this is true.

What is the "human preference," in this assertion?

My first thought was that as with Jev itself, it is not a position, it's a distribution of positions!

I think that human preferences, plural, will be a distribution of values across the population of various humans you ask; and, a set of specific conclusions humans will argue for on the basis of various philosophical strategies or maxims. Two different ways in which there is not any specific singular preference.

(I'm convinced you could not get uniform agreement on even "obvious" propositions, across a large sample set; and not necessarily just because of contrarian spoiling. Moral intuitions varying, is the why this problem is so well known, after all.)

I think the conclusion is partially right, but only if you adjust it to say, if what you want is alignment with a specific outcome, you need to be able to defined that outcome, even if you can't argue for it.

That's one of the fundamental problems of alignment, full stop; we can't from axioms construct the set of moral propositions we would like. We don't know what the right set of axioms is; we just know "right" when we see it. (And our neighbor may or may not see the same.)