Comment by hug
25 days ago
I see this kind of argument fairly frequently, and it just always seems like such a surface-level argument against prompting AI in this way.
This isn't a dig at you specifically, but the pithy answer to this kind of skepticism is, in a general sense: So what? I don't believe you have any of that either.
Obviously you & chatGPT aren't built the same, but in a practical-results kind of way in this scenario you are, because you're almost certainly unable to completely avoid your preconceived biases when asked any kind of complex question. You aren't aware of your subconscious biases, or how they're weighted against your overall thought process, and you can't tell me exactly what it is that happens when I ask you to try to ignore them. If we did some kind of implicit association test and found one of your subconscious biases, you may not even know how those biases came to be.
All of that to say: chatGPT can ignore its training as much as many people can ignore theirs: Not very well, but it'll certainly adjust the responses towards the thing you asked them to.
Clearly the prompting works, but the I think the more interesting question is why. Even from a just-get-things-done perspective, if you understand the mechanism of how and why a prompting technique works, that's going make you more successful in iterating on that technique in the future. IMHO that attempt to understand how and why your prompt works before you iterate on it is the difference between prompt engineering and prompt alchemy.
I agree that humans have the same limitation. I don't see the inability to dynamically remove training data as an LLM-specific problem.
For an LLM conscious thought is the tokens in its context window, and subconscious thought is the training embedded in its parameters. A person thinks in a similar manner, with subconscious gut instincts modulated (more or less) by a thin veneer of consciousness.