Comment by xyzsparetimexyz
2 hours ago
I have a pi extension that just runs the same prompt in a loop 25 times. I tried giving it a loop breaker but I found that it'd give up too readily. When the task is actually complete each iteration didn't do a lot of work so it was efficient enough. I suppose another way is to call out to a separate context to check if the task is complete?
Commands like /goal and similar are the more complex version of this; you write a prompt like it was a singular iteration, it runs one iteration, then runs an "evaluator" to determine if the goal has been reached, then runs the prompt again with a little extra to make it go again - and so on. The evaluator is just an LLM with most of the result or context looking at the original goal and the state and answering the question "has the goal been met".
The interesting part of this: while some leading implementations use the same LLM and context for the evaluator, some call out to a different context, some to a tuned LLM and different context; so which is better? many blog-scale benchmarks are calling it a toss-up that is highly dependent on the primary model.