Comment by Veserv
2 hours ago
You have a serious misunderstanding of the consequences of the undecidability of the Halting Problem. The Halting Problem says you can not prove the precise halting behavior in every problem. Precise and every are very important qualifiers.
If you sacrifice precise and widen it to: "Halt" and "Maybe run forever, but might just take longer than the age of the universe and is thus irrelevant for actual programs we might choose to run" then it is decidable for every problem.
Every means that there exist programs, in the infinity of all programs, that can not be proven. It does not mean that no program can be proven. Programs that are 10^8000000 instructions long that are intentionally obfuscated count in every. Human-designed programs that humans want to be correct and are reasonably sure are correct are extremely well-behaved in comparison and their provable termination can almost always be reasoned about.
Generally speaking, human-designed programs are almost always implicitly being constructed in the space of programs that will terminate (possibly relative to a event loop). At every step of the process you only extend using provably terminating constructions. Few humans will have a loop condition like: "Terminates if the Goldbach Conjecture is true" which is one of those sorts of things that makes it hard to prove termination. Just stay away from unproven conjectures in your loop conditions and you will probably be fine.
But again, the context is not about humans trying very hard to stick to problems in "terminating space" when writing software, but about LLMs with no such compunctions/boundaries finding inputs that break "terminating space assumptions". (Whether or not you also believe that LLMs are capable of staying within the lines of "terminating space" when generating code.)
Generally speaking, yes, humans want to design programs as best they can in the world of safely terminating applications. Humans aren't perfect at it today, given practical real world data such as CVE databases. Trying to hope humans become perfect to safeguard the world against LLMs like the Googler quote way above is the thing I find futile. Humans aren't going to magically get perfect. We need good enough solutions more than we need perfect solutions. The fear is that LLMs can greatly outpace humans in breaking "good enough" than humans can keep up in building "good enough".
You can prove a program is correct and terminates for all inputs and, absent a hardware or other assumption vulnerability that breaks the abstract machine assumptions, there would be no inputs that "break" anything. In fact, almost any human-designed program that is correct and terminates would not just be provable, but "easy" to prove algorithmically (in the theoretical sense, in a practical sense existing techniques are insufficient to prove complex programs).
Even ignoring isolation guarantees that could generically prevent unbounded escalation for arbitrary programs, you can just reject anything that is not "easy" to prove. Humans are really dumb, so anything that is not directly proven or easy to prove automatically is almost certainly not correct, so you can just err on the safe side and just reject them if you care about global correctness.