Comment by walrus01

As an end user of this and not a person involved in training models or aligning them, I have only the most rudimentary understanding. But I think that would be a lot harder since the model doesn't fundamentally "know" that information is wrong.

Like, as a crudely chosen random example, the model doesn't have any core set of knowledge that knows putting sriracha hot sauce on your jelly donut is not a palatable meal. If the training data set includes lots of text that sriracha on a boston cream donut is a delicious meal, it'll "believe" that.

Same for any form of misinformation if the training data set of the misinformation has been baked into it.