Comment by geraneum

11 hours ago

> Have it work on a task.

With or without a harness?

Have you actually tried this yourself? Of course it can derail it. Try to reflect on your interactions with LLMs without all the constraints like web search, agentic scaffolding, etc.

The same way that a “yes” or a “no” input from you can change the response, cot tokens are fed back into the model as input and can derail it.

5 comments

geraneum

Reply
user43928  9 hours ago

Have you? Can you show such a derailment with a large SOTA model?

It would be interesting.

I have seen such derailments within the GHCP harness maybe with GPT 5.6 Luna that went into some loop about whether it already provided a final response to the user, or 5.6 Sol suddenly switching to talking about MS SQL performance.

I also saw a post about Sonnet unexpectedly talking about Minecraft after seeing a file with a related name. The user thought it was the output of another user's conversation so the post was fairly popular.

  • geraneum  7 hours ago

    > Harness… seeing a file…

    Thank you for making my point for me. But let’s keep the goalposts stationary. We’re talking about LLMs without scaffolding.

    • user43928  4 hours ago

      Indeed, and it would be interesting whether it is much more likely to derail outside of a coding harness like in my examples.

      I still don't know if that is the case, and how frequently it happens, since you did not share details beyond vaguely suggesting it would happen.