Comment by imimayj1337

3 days ago

I know we just had the carbon announcement which was distinctly confusing, but comparatively I think that this is an exceptionally well-written article. I do believe that Google has some exceptional scientific minds as illustrated by their ability to take highly technical concepts and explain them in an understandable way for the layperson.

As for the technical nature of this, I'm not sure if I've totally understood correctly despite how well it's written, but are they essentially holding back sort of unweighted parameters built in to the model architecture which they can then populate with new context during 'sleep' and then using the techniques outlined in the 'dream' section to reinforce what gets stored in those MoE weights? That to me seems like a solution which only Google is best placed to solve due to their fully integrated AI stack and historic experience with hardware optimisation, but which would be rather hard to do if you are a more resource and compute constrained organisation

2 comments

imimayj1337

Reply
gavinray  3 days ago 

  > are they essentially holding back sort of unweighted parameters built in to the model architecture which they can then populate with new context during 'sleep' and then using the techniques outlined in the 'dream' section to reinforce...

Sort of. As the model is interacted with, short term "memories" are accumulated in a sort of buffer/cache. During "sleep" these "memories" are given a ranking/score, and the useful bits are integrated into the weights.

They say the weight update can be done by computing a transform where the end state is the old weights with the addition of the useful semantic knowledge contained in memories. Rather than just additively chucking the entire memory concepts into weights. That's pretty neat.

The subtext for the memory architecture diagram sums the whole thing up pretty tidily I think:

  > Input streams from the active phase are stored in short-term memory before entering the sleep cycle, which bifurcates into Consolidation (transferring fragile short-term tokens into expanded long-term parametric memory using low-rank structures and MoE) and Dreaming (autonomous self-improvement via synthetic routing with random experts and RL refinement).

  • sendgeetikamail  3 hours ago

    A very low tech version of this would be maintaining your md files after a work period. Down rank or delete irrelevant bits, mark as important and prioritize relevant bits. What's important is not all context that is important at the beginning of a work period remains important at the end of the period. It's this "learning" that needs to be captured.