Comment by c7b
19 hours ago
I wasn't aware that we have access to raw reasoning tokens? I thought what you get is a kind of summary. Does the author have some kind of privileged access or was my assumption wrong?
But for the question studied here it probably doesn't matter - overlaps in the publicly available output may be indicative of distillation (or not), regardless of what it is. I would just find it surprising that the Chinese labs would use it so trustingly. The publicly released reasoning trace is the first place where I would suspect some distillation poisoning to be injected.
They reference this paper which describes a method to decrypt reasoning traces (by sending the encrypted trace back to the model and asking it to transcribe it):
https://stolen-thoughts.com/paper.pdf
Interesting, but I suppose that's a hole that can be easily patched.
patched with gpt 6
2 replies →
They do store the reasoning locally. It's encrypted, though.
Few weeks ago there was a new paper out where researchers took the encrypted reasoning tokens and injected it into a new session with a week or model in the same family that they could reliably jailbreak. They would then ask the model to repeat its reasoning and the results were pretty consistent.
They used the LLM as a decryption oracle of sorts.
Reasoning works as long as there is a consistent latent space representation. Any kind of poison will just become part of the representation. There's evidence that even directly training on encrypted reasoning traces works, because the length is already a strong signal.
[flagged]