Comment by howunfortunate
11 hours ago
That's not quite true though. The fact that most models are in practice non-deterministic has been a huge point of contention.
And there's nothing inherently stopping labs from continuously fine-tuning the weights after every new invocation. It's just a difficult (not to mention expensive) software and ML problem.
I was not saying that they are deterministic, rather that the distributions (aka weights) are fixed. A model as deployed today at anthropic/open ai/etc is not learning beyond the context as far as I know.
What prevents continuous fine-tuning from what I understand is catastrophic forgetting. You can do things like RLHF which are built to minimize the damage but that is more about bringing out capabilities of the base model than incorporating new knowledge (at least from my understanding, I am obviously not a researcher at a lab).
But they routinely leverage web search in connection to providing responses which means they're leveraging their static intelligence on top of a dynamic context corpus. This is critically important and arguably similar to humans in other words the typical human might take a while to develop a new skill but they can change how they dynamically leverage an existing skill based on context
Yes, catastrophic forgetting is absolutely one of the problems that needs to be solved to enable something like this.
My broader point is just that there's nothing inherent to the structure of LLMs that stops them from updating their weights and continuously learning from environmental feedback in the way humans do, and there's already solid templates for how they could push even further in that direction.
But as an assessment of the current state, I agree with you, LLMs lag humans severely in ability to self-update.
LLMs are "readonly" I guess for several reasons:
1. Technical cost of updating the mode.
2. Inability to trust every user's "truth".
3. Ability of AGENT-HARNESSES to learn with the help of the human user.
So agents learn, LLM already knows everything it will ever know, and ESPECIALLY it has already learned how to understand human language.
No 3. above means there is no danger of the LLM getting corrupted. But the agents running on user's machine learn on behalf of that user who shares the machine with them.
>My broader point is just that there's nothing inherent to the structure of LLMs that stops them from updating their weights and continuously learning from environmental feedback in the way humans do, and there's already solid templates for how they could push even further in that direction.
"LLM" is a branded model as a product. Of course it could be anything, as long as it fulfills the product category.
But we live in reality, we can only look at what models are out there and we see that they don't do any of those things and yet we're supposed to act as if these models already do.
1 reply →
How are they non-deterministic? Ok there is random involved but for watermarking results the random is substituted by pseudo-random, otherwise it won’t work. If you control the randomness, you should be able to reproduce answers to prompts in equal context 100%.
It's not "not quite true", it's literally true because alternative architectures like RNNs and Mamba fully update their own internal states, whereas transformers only append to the context.
RNNs and Mamaba do not update their weights, but you could hypothetically scale the internal state to be as big as Fable's and GPT 6's parameters.
At least one mechanism to update transformers' internal states already exists, there is nothing stopping anyone from performing backprop after every session.
It just has big technical and economic challenges. But I expect advances there. There have actually already been big advances, though done in bulk fashion (RLHF).