Comment by wren6991

> I am told this is merely something in the system prompt that the model tends not to pay attention to with large contexts.

IIRC it's a system reminder injected after every single turn.

It must be pretty ingrained to be so resilient against prompting. I think RL on relatively short-horizon programming tasks has given the model a tendency to write down absolutely everything, so it survives compaction. Longer-term (project-scale) tasks where this crap starts to pile up and cause problems are in the evolutionary shadow, so to speak.