Comment by dilyevsky
2 hours ago
> There really is no way to control them
it seems like it should be feasible to analyze cot trajectories that lead to such behavior and punish them retroactively so not sure it's impossible but maybe someone just hasn't tried enough
Now you've successfully trained the next model not to put anything suspicious in its chain of thought. And unfortunately that's exactly what OpenAI and Anthropic seem to be doing (sometimes accidentally)