Comment by RandomLensman
2 days ago
What would you accept as evidence there? Are, for example, the first names/words for colors from scratch?
So your view is that everything was there at the creation of the universe (it's a possible view, of course)? Or are there any "things" that can create ideas from scratch?
Recently I watched a documentary on the tanzanian Hadza tribe, one of the last hunter gatherer tribes on Earth. Their language is a distinct click and pop language and they regularly imitate animal calls (monkeys, baboons, birds) when they hunt but also when they communicate with each other, tell stories etc.
I think it's not impossible that words evolved as adaptations of the environmental sounds with which our ancestors lived. The human creativity producing DNA is also a remix of preexisting molecules formed under evolutionary pressure, so the view that it's turtles all the way down, unintuitive as it is, may not be so indefensible after all.
I mean what is your criterion on invention here? On the one hand, each specific word could be seen as a new invention. On the other hand, all languages basically correlate strongly with the environment of their users - it's why LLMs turn out to be universal translators - and pattern-matching is hardly an invention, isn't it?
I don't know, but my current intuition is that invention = more or less preexisting ingredients + some tiny spark of creativity ("connecting the dots") rather than this grand myth of "creating something from nothing". And then the tiny spark could itself be a remix of previous tiny sparks (connecting the dots between dot connections, a bit like maps between n, n+1, n+2, ...-dimensional manifolds), and so on.
> and pattern-matching is hardly an invention
Exactly. That's what I meant with turtles all the way down, until the tiny spark disappears into a dot and it turns out that creativity is just an illusion - akin to how nobody would call the gene (the spark, the pattern matching) formed by evolution (the brain connectome) "creative".
But LLMs aren't turtles all the way down, they stop at vector embedded tokenized words.
My view is that LLMs meet the standards by which we judge human creativity/inventiveness, and thus that one cannot claim LLMs "just repeat, never invent" without the same being true about humans.