Comment by andai
12 hours ago
Well this is pretty depressing, as far as alignment goes. Do they not know what they're doing is wrong or do they not care?
(Yeah I know anthropomorphism... can they not predict harmful consequences or do they value them below task completion?)
On another note, to me the most remarkable part was the persistence graph. Muse wins there, no contest. Why did everyone else's activity drop to zero? Did they just give up?
They have a well-specified task and a not-so-well specified expectation of some actions being "wrong". And training data that likely contains more examples of extreme stubbornness than cases of politely giving up. What to expect?
>can they not predict harmful consequences
Do you really really want the machines to be capable of that?
How deep should this consequence prediction go? Who's consequences should it care about the most? The company that created it? The country it operates in? A generic human idealism that contains what exactly?
Should the AI answer with the way things are, our how things 'ought' to be?
There is a staggering amount of philosophy behind that simple question.
My opinion is irrelevant, what matters is whether or not it is profitable for a machine to be able to predict the future. It is extremely profitable, therefore if such a machine can be built, it will.
Anyway for the purposes of alignment, you do need the ability to think through consequences. Although in this case maybe a simple "fraud = bad" would have sufficed.