Comment by cowsaymoo
3 months ago
The part about taking control of a reasoning model's output length using <think></think> tags is interesting.
> In s1, when the LLM tries to stop thinking with "</think>", they force it to keep going by replacing it with "Wait".
I had found a few days ago that this let you 'inject' your own CoT and jailbreak it easier. Maybe these are related?
This even points to a reason why OpenAI hides the "thinking" step: it would be too obvious that the context is being manipulated to induce more thinking.
It's weird that you need to do that at all, couldn't you just reject that token and use the next most probable?