Comment by DoctorOetker

if language models (current GPT-style or any future model with different architecture) can be viewed as a compression of their output corpus (the corpus it emits by providing random starting native contexts weighted by the model's likelihood of that native context) then one is saying the corpus contains the same information bits as the model, the model just takes less data bits, while the corpus represents inflated data bits.

If learning requires communication of data, one could ponder if it is more effective to transmit learning data as suitable model coefficients instead of corpus monologue or dialogue (say with a teacher).

This may sound absurd, but is it really so far fetched to believe one could formulate "download"-compatible LLM architectures, so that the user or student can play a kind of memory game to train "download"-architected model weights by simple reflex games?

Suppose instead of token vectors we used token matrices or token multivectors as in geometric algebra (not to be confused with algebraic geometry!).

Word2Vec couldn't do general language modelling because it used vectors, but when using token matrices or token multivectors one could postulate the following partition function:

given a dictionary of tokens, and their corresponding (matrix/multivector) M one could define the partition function

exp(-|M1 * M2|^2)

For example it could be alphabet level (to demonstrate the concept):

For every string one can compute the corresponding M in terms of the elementary character matrices:

M_"car" = M_"c" * M_"a" * M_"r"

both matrix products as well as geometric multivector products are associative but not commutative:

(M_"c" * M_"a") * M_"r" = M_"c" * M_"a" * M_"r" = M_"c" * (M_"a" * M_"r")

But M_"car" != M_"rac" since matrix/geometric products is not commutative.

The result of non-linear but layer-less and order aware architectures could promise the following:

Feasibility of uploading model weights: a random grand tour of 2D projections of the square matrix space, projecting the token positions down to 2 dimensions along an changing axis, a "game" could correctly project 95% of the tokens, requiring the user to identify the incorrectly placed tokens, which jump back to their correct position when clicked.

This means a user can learn (since most of the time tokens are rendered in the correct position of the 2D projected cloud), and as a user learns their performance will go up.

If a user could eventually (after playing for a long time) correct 90% of the token positions on random 2D projected planes, then the user has effectively stored 90% of their coordinates.

Suppose such a user reads text in their mother tongue, then the joint likelihoods in natural text will correspond to matrix products of those weights. In other words the brain will learn that whatever it has learnt playing the token-game aids the brain in predicting a next token.

Once it has learnt to utilize the format (matrix / multivector coordinates), the apparently pointless skill of positioning tokens in the cloud learnt during the game, it would also help predict the next token in languages the user didn't know, say when looking for ingredients on a product package. I believe such a user would swiftly discover they actually know those languages, and I believe transmission of a compressed format would take less time than transmission of the same knowledge in expanded output-corpus form.

I wonder if Terrence Tao has an opinion on such a possibility:

1) does he consider it (im)possible to have token-first formatted LLM's without MLP layers etc reach similar levels of performance as the current crop of LLM's?

2) supposing it were possible, would he predict a user "downloading" weights to his brain by a reflex-game as feasible?

3) does he believe a human brain would be able to make use of the downloaded weights and would the brain notice the utility in predicting tokens?

4) does he believe that once the brain has noticed / learnt how to make use of the model weights, first for predicting the next token in the mother tongue, it would subsequently have learnt how to predict in other tongues?