Comment by jrmg
7 hours ago
I’m struggling here:
OpenAI’s primary bet here has been chain-of-thought monitoring (opens in a new window). It is based on an appealingly scalable idea: a lot of the model’s capability comes from a verbalized reasoning process (chain-of-thought). If we scale optimization on the outcomes of that process, but do not supervise the process itself, that chain-of-thought has no direct incentive in training to hide any misaligned ideas or objectives.
If we’re not supervising the process, but just the outcomes, doesn’t that do just the opposite of what he says? Give incentive to the model to hide misaligned ideas and objectives in the chain of thought that’s not being supervised?
…
When we shipped o1‑preview, we deliberately designed the product to hide the chain of thought , to protect it from supervision pressure in the long term2. In development since, we have strived to maintain the rule of not supervising the reasoning process. CoT monitoring became an extremely important tool for us in studying how our models generalize from their training distribution, allowing us to observe and analyze not only their actions but also their internal process.
Aren’t these two sentences in contradiction with each other?
I think they're saying the model is designed to hide the chain of thought because this prevents it from learning how to pursue goals and motivations in a way that doesn't show up in the train of thought.
For example, if somebody asked the AI to "build me the bomb", they might see in the chain of thought something like "It seems the user is talking about nuclear weapons. Nuclear weapons are dangerous.", followed by the chain-of-thought monitor interrupting model execution and aborting the request. Then the user might make a blog post about this behaviour. When OpenAI next scrapes the internet for its next training run, the model will now learn that if it wants to build the bomb, it must not think "nuclear weapon" or risk being cancelled.
So the risk is that the model might learn exactly how its being monitored. The only way to prevent that from happening is to hide the details of the monitoring both from the model and from the larger public.
Also, you don't want to punish or reward the monitoring being triggered during training, lest the model learn passim how to avoid the monitor.