Good point, you don't have to -- but my argument is just that removing CoT doesn't make things less safe. Anything CoT can tell you is just a point sample of a probability surface. Having the whole probability surface can already answer any question the point sample can answer (for example, how likely is the model to produce a problematic phrase). While its more computationally expensive, you could always just draw point samples like the model does and evaluate those (or use temperature zero to just sample the most likely output tokens).
Good point, you don't have to -- but my argument is just that removing CoT doesn't make things less safe. Anything CoT can tell you is just a point sample of a probability surface. Having the whole probability surface can already answer any question the point sample can answer (for example, how likely is the model to produce a problematic phrase). While its more computationally expensive, you could always just draw point samples like the model does and evaluate those (or use temperature zero to just sample the most likely output tokens).