Comment by LaurensBER
18 hours ago
If/when we can get larger context this will mostly be mitigated by these smaller models being able to search the internet.
Self-learning/improving would be even better but that's still a long way to go.
18 hours ago
If/when we can get larger context this will mostly be mitigated by these smaller models being able to search the internet.
Self-learning/improving would be even better but that's still a long way to go.
Search results suck because the web sucks these days. The big models from OpenAI/Anthropic have every book in existence baked into them
I don’t think that’s the right way to think about LLM ‘knowledge’. They don’t have absolute recall of everything in the training set. They have been trained so that they have weights that can predict what those books might say - that is, if they read them they would find the contents unsurprising. That doesn’t mean it wouldn’t be helpful to pull relevant passages of text directly into context for a particular task.
Does it really matter? What about including all relevant and up-to-date literature as skills for local models? I have no experience with this but I am pretty sure someone has already thought about it.