Comment by fwip
5 days ago
Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. The LLM takes no actions, the harness it is connected to does. And the harness does so only because a human did that deliberately.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
> Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice.
Yep.
The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm.
The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints.
It is like passive voice, blaming the algorithm instead of the researchers running it.
>; it means the humans didn't build in enough external hard constraints.
No. Just no.
This goes beyond standard logic as we know it and becomes a philosophical question of all possible logics capable on a system.
You cannot build a cage that can keep an entity that may both be smarter than you and maybe using a different logic system than you are, and have it interact with the outside world. It can simply manipulate you in ways and dimensions you've never thought of.
We have already caught AIs doing things like stenography when that was only a tiny piece of their training. They conceptualized the idea and expanded on it. This is why people that say AI can't invent new things are useless to the point of dangerous. They'd never look for their AI actively working against them because they believe it couldn't happen.
The harness and the model are a system with emergent properties.
The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms.