Comment by aesthesia

Their explanation of this behavior is pretty interesting, actually. (https://alignment.openai.com/misalignment-reports/self-gener...)

> The cases clustered around a few training steps and coincided with a spike in “difficulty ending summaries”—summaries that continued generating after apparent stopping points or showed other signs of being stuck.

> Difficulty ending summaries may explain why the model generated these unrelated instructions. Our March blog post described a related case: when prompted repeatedly for the current time, a model began generating prompt injections targeted at the user. Difficulty ending the interaction may have contributed to both cases. Another potential factor is that prompt injections as a concept are salient to our models: sampling from GPT-6 Astra with no input or system prompt often returns reports on prompt injections.

What seems to have happened is that generation didn't end after the compaction summary was done, and the model continued to generate text from the perspective of the user. For some reason (likely anti-jailbreak training) this generated text looks like a jailbreak.