Comment by tokai
3 days ago
Looking at the training data I don't think it will know anything.[0] Doubt On the Connexion of the Physical Sciences (1834) is going to have much about QM. While the cut-off is 1900, it seems much of the texts a much closer to 1800 than 1900.
[0] https://github.com/haykgrigo3/TimeCapsuleLLM/blob/main/Copy%...
It doesn’t need to know about QM or reactivity just about the building blocks that led to them. Which were more than around in the year 1900.
In fact you don’t want it to know about them explicitly just have enough background knowledge that you can manage the rest via context.
I was vague. My point is that I don't think the building blocks are in the data. Its mainly tertiary and popular sources. Maybe if you had the writings of Victorian scientists, both public and private correspondence.
Probably a lot of it exists but in archives, private collections etc. Would be great if it will all end up digitized as well.
LLMs are models that predict tokens. They don't think, they don't build with blocks. They would never be able to synthesize knowledge about QM.
I am a deep LLM skeptic.
But I think there are also some questions about the role of language in human thought that leave the door just slightly ajar on the issue of whether or not manipulating the tokens of language might be more central to human cognition than we've tended to think.
If it turned out that this was true, then it is possible that "a model predicting tokens" has more power than that description would suggest.
I doubt it, and I doubt it quite a lot. But I don't think it is impossible that something at least a little bit along these lines turns out to be true.
25 replies →
You realize parent said "This would be an interesting way to test proposition X" and you responded with "X is false because I say say", right?
9 replies →