Comment by andai

5 hours ago

> Now, here's where our fun experiment starts. I found out that by setting the gate to 1,1 the write strength (beta) to 1,2 and using one-hot key encoding,3 every write we perform will overwrite exactly one row, and every one-hot query can retrieve an exact value. This means that the GDN layers themselves can work exactly as RAM, at a far lower cost than using normal attention as RAM, since a GDN head's state stays the same size however long the conversation gets, unlike attention's KV cache.

So like... can't you use a computer as a computer without doing all that roundabout stuff? Like if you want to integrate a LLM with hard computation there's probably a more efficient way to do that.

Or is this building towards "but can it run DOOM?" :)

5 comments

andai

Reply
rao-v  5 hours ago

I’ve always wondered why we don’t add a few frozen “layers” to models that perform complex computations 2/3rds deep in the model. Some set of steerable fft + some common vector math operations could be reusable units of capability that the model learns to leverage during training.

Given that models learn to use residual connections optimally, it can’t be hard for them to steer around the fixed computation layer when it’s not needed.

I’m sure giant models invest a few layers in reproducing these functions, but surely smaller models can benefit?

  • zxexz  34 minutes ago

    I’ve done it by hardcoding a knowledge graph nn and adding a few “query” layers. You can freeze the graph and swap it out and update it during training so you can update the knowledge at will during inference. You just need to keep the format the same; the model learns an arbitrary graph query language. If you register a buffer it can update you can teach it to keep scratchpad memory very quickly.

vavos  1 hour ago

I think these underlying theory/hope it would lead to better integration of the computer/calculator into the model. Similar to mental arithmetic vs using a calculator. mental arithmetic is way smoother in the thinking process if you're good at it ( which putting a small computer inside your brain would probably achieve)