Comment by kqr

I noticed this when I tried to get LLMs to play text adventures. Early on in my experiments, I wanted to give them hints when they got stuck at a puzzle. I did this by stopping the loop, injecting thoughts into the LLMs own persistent scratchpad (as if it had thought of that itself), and then starting the loop up again.[1] That way, the LLM would read what it had intended to remember from the previous turn including the hints I injected, and use that for the next turn.

I discovered that LLM-generated tokens in the scratchpad were relatively stable, but injected thoughts were frequently ignored and often deleted from the scratchpad within a few turns – even when the injected thought was the literal answer to the puzzle it was stuck at!

A reader[2] then pointed me toward research similar to what you might recall: LLMs interpret text by maintaining activations for input tokens, so text that is not generated by the same LLM will seem "unlikely" to the LLM in a sense, and when given the alternative between likely and unlikely text, it's probably trained to judge the unlikely text as a weird "slip of the mind" and discredit it in favour of the more likely text. I speculate this is part of how they can be useful in the first place, despite their non-determinism.

[1]: https://entropicthoughts.com/getting-an-llm-to-play-text-adv...

[2]: https://entropicthoughts.com/getting-an-llm-to-play-text-adv...