Comment by onraglanroad
1 day ago
So you have two AIs. Let's call them Claude and Hal. Whenever Claude gets something wrong, Hal is shown what went wrong and asked to rewrite the claude.md prompt to get Claude to do it right. Eventually Hal starts shouting at Claude.
Why is this inevitable? Because Hal only ever sees Claude's failures and none of the successes. So of course Hal gets frustrated and angry that Claude continually gets everything wrong no matter how Hal prompts him.
(Of course it's not really getting frustrated and annoyed, but a person would, so Hal plays that role)
I don't think it's inevitable often the AI will just keep looping again and again. It can happily without frustration loop forever.
It doesn't loop though -- it has continuously updating context -- and if that context continues to head one direction it will eventually break down.
My own personal experience with LLMs is that after enough context they just become useless -- starting to make stupid mistakes that they successfully avoided earlier.
I assume old failures aren't kept in the context window at all, for the simple reason that the context window isn't that big.