Comment by bwest87
11 hours ago
He forgot the tokens!
It's not simple weights and numbers all the way down. The available output is pre-set by the tokens we allow it to predict.
There was a whole bit in there about not having a language module or using words. But it does. We tell it.
Humans do not come pre programmed with a set of possible "tokens". We just figure it out and I believe that fact captures something very essential. Maybe the missing piece of AGI. The fact that humans can just be awash in pure sense data, and somehow just figure out what is important and what to do. Never ceases to amaze me.
The set of tokens is learned, more or less. So I don’t get what point you’re trying to make here. There’s not a human manually deciding what tokens make up the token dictionary.