Comment by njstraub608

The types of systems from the paper get more interesting when you’re dealing with really tight voice-based conversation loops. If the workload isn’t latency sensitive and you can just have a harness execute the types of searches you mentioned it’s rarely worth it to explore these types of optimizations or memory structures (or using embeddings at all). However, if I ask a model for something we talked about yesterday I don’t want to have to wait for it to reason and then make a tool call then read it and reason again. Even the, “let me check on that” gets tired quickly. They mentioned a 27x speed up putting the retrieval at 0.79 seconds vs 19s which is very promising for voice-based personal assistants.