Comment by sigpwned
5 days ago
The big questions I’m taking away are:
(1) they are claiming to produce apparently bijective closed-form symbolic representations/approximations of, among other things, LLMs. Is evaluating these closed-form representations more computationally efficient? The implications of that are potentially huge. It would be essentially analytic distillation. Fable on a chip and not a data center would be important — and disruptive - in many ways.
(2) Unsupervised, and even supervised, symbolic approaches to problem solving break down due to combinatorial explosion, among other things. This could potentially allow us to treat LLM training and inference as a search algorithm for novel symbolic approaches to solving new classes of complex problems hitherto unreachable through other approaches. If that works, I suspect it’s a feedback loop, too - the learnings from one representation push advances in the other. This would also increase the economic value of large training runs, since the model itself is now valuable, not just its inference.
(3) Per the above, can this push LLM design to greater capabilities?
The relationship between this and Anthropic’s J-space observation is also interesting. This is much, much deeper and more directly actionable, though.
EDIT: I ran my questions through Sonnet — yes, I appreciate the irony — and it was none too sanguine about questions (1) and (2), but thought (3) was reasonable. In any case, this is quite the paper. On reflection, I do think that the apparent reliance on very simple symbolic representations and tasks is underwhelming. But the approach is impressive. And obviously this is still early days, and the value of building a bridge between the very fuzzy LLM models and the rigorous, mechanically provable models would be enormous.
Page 7.
They train what appears as embeddings for outer product of roles and fillers. The role for language model can be a position in text, the filler can be an embedding of a word at that position. Then that matrix of a sum of these outer products is linearly mapped into NN encodings and then decoded by NN decoder.
The embeddings learned by this process are not necessarily smaller than original ones. Given that they participate in an outer product computation gives me impression that the resulting sum is much bigger than actual NN encoding, that is why it needs to be linearly mapped into NN encoding.
So, this paper will not necessarily lead to any computation savings.
But I am at page 6. ;)
Good find! But they stop short of saying it cannot be distilled to symbolic algebra. Regardless, your point stands. I scanned the paper in the middle of the night instead of sleeping. Clearly I didn’t do a close read! Thank you for pointing that out.
Their representation is a key-value store that they embed via a tensor product to reconstruct the vectors the model produces. The catch is that it works best if the key-value mapping is task-specific. So if you already know how to solve the task the model is performing and can transform the input into a data structure that makes it easy to just read off the answer, you can probably also recover the model output.
That in itself is of course not super useful for tasks that we don't already know to do symbolically, but it's nice to see that they can manipulate the keys and values of their representation somewhat independently. So at least some of what LLMs do internally seems to involve a bit more structure than just linear combinations of some overcomplete dictionary.
The interesting question is whether this can be used to interpret the operation of a single layer by lifting it to operate on key-value stores instead, replacing matrix-vector products with reading from some keys and then writing to others. That could potentially move the balance of power more towards CPU inference instead of GPUs.
> So at least some of what LLMs do internally seems to involve a bit more structure than just linear combinations of some overcomplete dictionary.
Well that's good to hear. Would have been rather exasperating to find out we could have somehow gotten Fable or Sol out of sufficiently advanced sparse-coding ages ago.
Maybe someone can clear this up for me. I always thought it was already obvious that neural networks, artificial or otherwise, are symbol systems. Some input pattern reliably associated with some output pattern is, nearly by definition, a symbol. No? Am I just using the word symbol differently than researchers in these fields?
> Is evaluating these closed-form representations more computationally efficient?
No it is way worse since computationally they are same things with symbolic having overhead of attending to the symbols.
My naive middle-of-the-night Claude question said the same thing. At least for this approach. I have not read the paper closely enough to refute you. But the concept of a lower-dimensional closed-form solution — which the paper seeks to discuss, please correct me if you read differently — is tantalizing, if only because it opens the path to different math, which can lead to optimization.
Is it though? Brute force searching generally scales like 2^n. LLMs on the other hand can obviously be trained, and even though training is massively resource intensive, it's not 2^n.
It's a fascinating emergent behaviour but also one that could have been predicted?
I'd imagine that our brains have become similarly biased towards generating such a generalised symbolic structure over hundreds of millions of years of evolution?
I'm thinking how certain regions of the brain reliably represent things like motor skills, speech, abstract thought.
I agree. In retrospect, this seems almost inevitable. And our own minds at least to do some form of symbolic reasoning — literally language, which you are apparently capable of, dear reader, as a lower bound. There might be more symbolic reasoning in the conscious, unconscious, and subliminal parts of the mental workspace. I always hesitate to infer similarities between the brain/mind and LLMs, but I certainly track the comparison here.
Inference is just tokens transformed through a fixed crystalline structure, no? You already could put that on a chip. There's no particular reason it couldn't be represented as some set of symbolic equations instead of a layered process... it's just another kind of quantization.
When symbolic algorithms are that large, they're equally hard to reason with most of the time. The upshot would be a lot more storage required in exchange for more generalized computing, lessening the need for so much GPU in a lot of cases.
I don't see why a model couldn't be represented that way. After all, if you just polled the output of a model, you could evolve genetic algorithms to predict it with fairly high accuracy in a limited domain. Take that out to the Nth degree and you're basically just unspooling the model into a giant set of equations.
Yes, you certainly could put that on a chip. And people are doing it, for smaller models. The question is how big that chip would be for something like Fable, which is generally estimated in the trillions+ of parameters, and if, given the amount of memory, computation, and bandwidth required - at least with current methods, i.e., very high dimensional matrix math — a chip is the right way to go. Can you put a LM (just a smaller language model) on a chip? Empirically, yes, I believe it is done. Small enough ones probably even fit on a FPGA. Can you put a LLM on a chip? Depends on how L it is! My intuition says that some open smaller models might fit, like Haiku, but not Fable. But that’s just intuition talking.
The hope would be that this unlocks some substantially more efficient or parsimonious math that would fit better on a chip. And that’s clearly my words, not the authors’, per the comment above.