Comment by komali2
18 hours ago
As I understand it it's undetermined yet whether LLMs can actually come up with anything novel or are instead pulling from their incredibly deep corpus of knowledge to present solutions that were there but we didn't realize it because our brains aren't libraries of almost all human writing.
Synthetic data allows them to train well past the limits of human writing.
What's an example of synthetic data?
Only in the same sense it's not yet determined about humans, either.
Not so sure. Was everything already "there" before humans existed?
In some form and shape, yes. Humanity's creativity is a lot of marginal copying and remixing.
But obviously, it adds up to something greater than went in; in aggregate, our contributions are something to awe.
But my point is, if you zoom in at the marginal, incremental contributions of any individual human in this process, it's really hard for me to say LLMs are not at the same level already.
On this topic, people like to compare LLMs to Einstein, but as far as I know, Einstein did not zero-shot special relativity in an afternoon. He built it up incrementally over time, it took him three times longer than the time between first ChatGPT release and today, and it depended on centuries of prior art, culminating in the right observation and right notation being available to him in his moment of greatness.
8 replies →
[dead]
No this is not an issue. As long as their is a way of verifying things, they do the same thing with creating novel things as humans: Searching through an infinite space of possibilities opitmized by knowledge.
They combine things, verify it and if it works and progresses the problem, they created something new.