Comment by kmeisthax

We're not trying to keep the AI from learning general ideas, we're trying to keep it from memorizing specific expressions[0]. There's a growing body of research to show that these models are doing a lot of memorizing, even if they're not regurgitating that data. For example, Google's little "ask GPT to repeat a word forever" trick, which will make GPT-4 spit out verbatim training data[1].

If there was a training process that let us pick a minimal sample of examples and turn it into a general purpose art generator or text generator, I think people would have been fine with that. But that's not what any of these models do. They were trained on shittons of creative expression, and there's statistical evidence that the models retain that expression, in a way that is fundamentally different from how humans remember, misremember, adapt, remix, and/or "play around with" other people's creativity.

[0] You called these "embodiments", but I believe you're trying to invoke the idea/expression divide, so I'll run with that.

[1] Or at least it did. OpenAI now filters out conversations that trip the bug.