← Back to context

Comment by tym0

3 days ago

> does not bloat up the context window too fast with reasoning tokens

How much does that matter if it's reset at every turn?

10 comments

tym0

Reply
jolux  3 days ago

what do you mean by reset at every turn? context stays until compaction. if you remove the reasoning tokens after every turn you will be constantly blowing cache which is far worse than filling up context.

  • tym0  3 days ago

    That's not my understanding of how most agents work. This is what a chain of request/response looks like:

      Your Prompt 1: Prompt Content 1 -> cache-1
  LLM Response 1: <Thinking>Thinking Content 1</Thinking> Response Content 1
  Your Prompt 2 (client side): prompt-1 + response-without-thinking-1 + Prompt Content 2
  Your Prompt 2 (server side): cache-1  + response-without-thinking-1 + Prompt Content 2 -> cache-2
  LLM Response 2: <Thinking>Thinking Content 2</Thinking> Response Content 2
  Etc...

    So reasoning gets dropped from context and you still get cache from the accumulating requests.

    Edit:

    I've realised I was incorrect, the thinking doesn't get passed back and forth but the latent snapshot does which result in using memory just the same.

    • mike_hearn  3 days ago

      Modern protocols loop back the reasoning tokens in raw textual form via an encrypted parameter. You can't see them (modulo the recent attack), but you do resubmit them.

      2 replies →

Alifatisk  3 days ago

Does it reset at every turn? From my experience in Codex for example, Luna (Max) fills the 256k token window relatively quick. The only thing lowering the context window again is the compaction.