Comment by napoleoncomplex
6 hours ago
This is the second ChatGPT shared conversation I've seen today that is truly fascinating.
The first one was someone proving another conjecture false by just repeatedly saying "keep going" to ChatGPT: https://x.com/DmitryRybin1/status/2079904005652893709
What a world we live in.
> just repeatedly saying "keep going" to ChatGPT
For posterity, this indeed works for most problems where an agent might give up. LLMs don't inherently know something is impossible.
The phrase I tend to use in my harder prompts to automate this with a sane loop breaker:
> **REPEAT THIS PROCESS UNTIL CONVERGENCE AND YOU ARE OUT OF OPTIMIZATION IDEAS.** You have permission to keep iterating.
I have a pi extension that just runs the same prompt in a loop 25 times. I tried giving it a loop breaker but I found that it'd give up too readily. When the task is actually complete each iteration didn't do a lot of work so it was efficient enough. I suppose another way is to call out to a separate context to check if the task is complete?
Commands like /goal and similar are the more complex version of this; you write a prompt like it was a singular iteration, it runs one iteration, then runs an "evaluator" to determine if the goal has been reached, then runs the prompt again with a little extra to make it go again - and so on. The evaluator is just an LLM with most of the result or context looking at the original goal and the state and answering the question "has the goal been met".
The interesting part of this: while some leading implementations use the same LLM and context for the evaluator, some call out to a different context, some to a tuned LLM and different context; so which is better? many blog-scale benchmarks are calling it a toss-up that is highly dependent on the primary model.
I’m a bit surprised OpenAI isn’t finding these big results far faster than the product’s user base. With no limits on runtime, access to dev models, custom tuning, and top talent, you’d think there’d be a constantly running internal project with the goal of solving famous math problems. And who knows, perhaps there is, but it would be interesting to compare the rate of success per unit “effort” of the internal mathematics work with that of the user base.
What I am thinking is the way you make it 'keep going' and when you have people of the calibre of Tao doing it I kept thinking how many breakthroughs is he going to cause the LLM to find with his targetted questions :D Amazing that we have the privilege of witnessing a true expert in such a way question the LLM.
Do notice he is quite Socratic, the approach works well for LLMs they love to please so you have to be careful in how you lead them.
this is /goal in claude code/codex. also basically a slightly improved ralph loop
Aka /goal
Without any more context, "keep going" seems to be doing a lot of work. The user is placing a lot of faith in the LLM to not make subtle logic mistakes and to take good approaches to each problem. In my experience, even frontier models (such as Fable) are quite capable of getting confused during even simple technical work I've done in the dev ops world. For example:
LLM: This package hasn't made it to production.
ME: are you sure? i see it right here!
LLM: You're right to push back. I inferred that based on weak data. I see now that the package has been deployed!
If the above conversation is typical for me, how could one expect to achieve a sound result by repeatedly prompting an LLM to simply "keep going" in dense mathematical proofs? Perhaps the user in this case had actually checked the LLM's work before issuing the prompt, but I think you see my point anyway.
Agreed, often you have to step in and stop it from reasoning itself into dumb directions, but occasionally it goes just like the transcript in question.
There may be something(s) about mathematics (proofs) that makes it particularly amenable to LLM reasoning - highly inductive from facts that are explicitly within-context/associative space? Being an unusually well documented discipline in general, with less influence from tacit knowledge or idiosyncratic “it works however the opinionated human made it work +- bugs” processes? Something about simulating even the smallest non-pure-inductive leaps necessarily risking simulating mistakes due to the nature of context “perception”?
There’s also probably a lot less noise from casual internet conversations. I imagine a nontrivial amount of what LLMs know about certain technologies comes directly from forums like reddit where quality of response isn’t guaranteed.
“Keep going” is exactly how many mathematicians achieved success in the past. :)
Is this the same as Dinitz Theorem[1] which seems to have been proved in 1994? This is the only result I keep stumbling upon when trying to understand the problem formulation
[1]: https://en.wikipedia.org/wiki/Dinitz_theorem
"it's enough of partial results. let's finish with a complete unconditional counterexample"
"Worked for 88m 24s... >"
"<h1>Complete finite counterexample</h1>"
...
crosses fingers "Low hanging fruit, low hanging fruit, low hanging fruit..." hyper-ventilates
Direct link: https://chatgpt.com/share/6a60b2eb-0b64-83ee-9c76-7931ca1de0...
From the prompt:
> You should do a breakthrough
This is just as funny and ridiculous as those "make no mistake" prompts.
Mathematical breakthrough, genius, trending on Artstation.
"whats next" is another good one
"Gew on, lad!"
this sounds like like an open parenthesis (
without someone independently verifying it, it just dangles there
...
At Mozilla, we had a set of whiteboard tags we could set on bugs, like "[crash]" or "[compat]" or "[leave-open]". That last was used when there were multiple patches attached to the bug, and we wanted to land only some of them without automation closing the bug once they landed. (It's common to have alternate approaches or test cases also attached to the bug, so you normally don't want to wait for all of them to land before closing the bug.)
I started using "[leave-open" for those.
It lasted for a couple of years, until someone went through and "fixed" them all.
)