Comment by forlorn_mammoth
7 hours ago
Hey, about that high dimensional space, is it continuous or discrete?
Also, I'm curious what you mean by "embed", the word implies a topographical mapping from "words" to some "high dimensional space". What are the topographical properties of words which are relevant for the task, and does the mapping preserve these?
circling back to the first point, are words continuous or discrete? is the space of all words differentiatable?
Discrete. But my understanding is that for all intents and purposes it is differentiable.
None of this means that you can infer the input space (human brain) from the output space (language). You can approximate it. But you cannot replicate it no matter how many weights are in your model. Or how many rows you have in your dataset. And it’s an open question of how good that approximation actually is. The Turing test is a red herring, and has nothing to do with the fundamental question of AGI.
Unless you have access to a Dyson sphere where you can simulate primate evolution. Existing datasets aren’t even close to that kind of training set.