Comment by jkingsman
1 day ago
So if I'm reading this right, it's basically a specialized embedding of a RAG, and then a model in the middle takes the agent's queries and returns responses in a form that is injecting into the main model's input token stream directly i.e. skipping actual text. And the injection format is INCREDIBLY token efficient. Sort of injecting a vibe rather than quotable text. Their results of "good on concepts, less good on quotes/exact text retrieval" seem to intuitively line up with that.
I was just trying to find work about this yesterday.. Wondering, if we have multimodal image,text,sound models, why not multimodal, text,graph models that can store their memory in more efficent format than text in their context, without needed external tooling..
agree
It looks like another difference with RAGs is that instead of reranking the doc vectors and taking the top 5 or so, this paper mixes all the docs (weighted by relevance) into a fixed number of "memory tokens".
It looks like this helps avoid a problem I've had with RAG - when a fact isn't local to a single patch of text.
Is this a problem, though? Vector similarity should surface the relevant, related patches.
But only in documents that make it through RAG’s hard cutoff. There’s an explanation of this on the related Stanford site at https://memory.stanford.edu/ (see “How MEMORILLA differs from RAG”)
[flagged]
I think its offering a compromise between accuracy and tokens. You don't have to use as many tokens and you get pretty good accuracy, but not the absolute best. I think, however, in many situations people aren't super concerned about the tokens as they are with the accuracy.
[flagged]