Comment by rcxdude
15 hours ago
I think neither the 'any set of morals is equally correct and workable' nor 'there is one fixed set of morality that is universal across time, place, and culture' is really defensible. It seems like morality is a set of ideas much like any other, and subject to selection pressure: some morals are more capable of surviving and propagating than others, and the common features we see across cultures are representative of that. But also we might expect that they shift with the structure of society and that has changed drastically because while human genetic evolution is progressing on a much longer timescale than recent history, memetic evolution is far faster (though for things like morality, still at generational timescales).
There are cross-cultural differences in moral reasoning, and I think this is very interesting. But that seems to make training a hyper-rational AI, with no meaningful theory of mind, to be 'moral' or 'just' even harder.