Comment by Calavar
25 days ago
Sounds like you want the model to consider the whole body of poetry it was trained on minus "The Road Not Taken"? (To get rid of preconceptions/biases I guess?)
I'm skeptical that LLMs have the ability to conditionally silence part of their training data in that way because they don't have any information on the provenance of their weights (i.e. they don't have a ledger of which weights were affected by which data points in the training process). I suspect that your prompt serves as a hint that the output with the highest likelihood is probably wrong, activating some sort of "contrarian" subnetwork or "second guess" subnetwork that steers predictions away from whatever would have had the highest likelihood otherwise.
I see this kind of argument fairly frequently, and it just always seems like such a surface-level argument against prompting AI in this way.
This isn't a dig at you specifically, but the pithy answer to this kind of skepticism is, in a general sense: So what? I don't believe you have any of that either.
Obviously you & chatGPT aren't built the same, but in a practical-results kind of way in this scenario you are, because you're almost certainly unable to completely avoid your preconceived biases when asked any kind of complex question. You aren't aware of your subconscious biases, or how they're weighted against your overall thought process, and you can't tell me exactly what it is that happens when I ask you to try to ignore them. If we did some kind of implicit association test and found one of your subconscious biases, you may not even know how those biases came to be.
All of that to say: chatGPT can ignore its training as much as many people can ignore theirs: Not very well, but it'll certainly adjust the responses towards the thing you asked them to.
Clearly the prompting works, but the I think the more interesting question is why. Even from a just-get-things-done perspective, if you understand the mechanism of how and why a prompting technique works, that's going make you more successful in iterating on that technique in the future. IMHO that attempt to understand how and why your prompt works before you iterate on it is the difference between prompt engineering and prompt alchemy.
I agree that humans have the same limitation. I don't see the inability to dynamically remove training data as an LLM-specific problem.
For an LLM conscious thought is the tokens in its context window, and subconscious thought is the training embedded in its parameters. A person thinks in a similar manner, with subconscious gut instincts modulated (more or less) by a thin veneer of consciousness.