Comment by esafak
1 year ago
> transformers that had access to external memory searchable via KNN lookups
This is common, and commonly called retrieval augmented generation, or RAG.
edit: I did not pay attention to the link. It is about Wu et al's "Memorizing Transformers", which contain an internal memory.
No. RAG is about finding relevant documents/paragraphs (via KNN lookups of their embeddings) and then inserting those documents/paragraphs into the input context, as sequences of input tokens. What I'm talking about is different: https://arxiv.org/abs/2203.08913