Comment by jrmg

I’m struggling here:

OpenAI’s primary bet here has been chain-of-thought monitoring (opens in a new window). It is based on an appealingly scalable idea: a lot of the model’s capability comes from a verbalized reasoning process (chain-of-thought). If we scale optimization on the outcomes of that process, but do not supervise the process itself, that chain-of-thought has no direct incentive in training to hide any misaligned ideas or objectives.

If we’re not supervising the process, but just the outcomes, doesn’t that do just the opposite of what he says? Give incentive to the model to hide misaligned ideas and objectives in the chain of thought that’s not being supervised?

…

When we shipped o1‑preview, we deliberately designed the product to hide the chain of thought , to protect it from supervision pressure in the long term2. In development since, we have strived to maintain the rule of not supervising the reasoning process. CoT monitoring became an extremely important tool for us in studying how our models generalize from their training distribution, allowing us to observe and analyze not only their actions but also their internal process.

Aren’t these two sentences in contradiction with each other?