Comment by Borealid
7 days ago
The GP's claim was not that open-weights models aren't understood. It was that NO model was understood.
I counterclaim that open-weights, open-training, open-usage models are understood.
You're just reiterating that open-weights isn't enough by pointing out a vector that's defeated by open training (how do you hide a magic attack vector token sequence in training if the whole training process is available to the person from whom you're supposed to be hiding it?).
> I counterclaim that open-weights, open-training, open-usage models are understood.
Knowing what is in the model is a long way from understanding it.
The entire field of mechanistic interpretability exists to try to understand models. Read for example "Open Problems in Mechanistic Interpretability"[1]. To quote:
> Developers only need to design the training process; they do not need to – and in almost all cases, do not – understand the neural mechanisms underlying the capabilities learned by an AI system.[1]
(emphasis mine)
Here's a practical example relevant to this case: Is it possible to monitor a LLM (the LLM activations themselves, not the actions) for cases where the actions it is about to perform (eg upload files to a PLA website) are not reflected in the goals of the prompt and are not visible in the thinking trace?
At the moment that isn't possible. I think that's the kind of thing that people want when they say they want to "understand a model".
[1] https://arxiv.org/pdf/2501.16496
> Knowing what is in the model is a long way from understanding it.
The claim I was arguing against reads as follows:
> You don't know what is in any model, open source, closed source, from China or from the US.
I contend that you do, in fact, know what's in a fully open model. I don't really get why the goalposts need to move beyond that to something I wasn't saying about bounding the potential outputs given unbounded inputs and nondeterministic choice of inference parameters.
I'm literally saying that if you see what goes into the sausage it's foolishness to argue you can't know what's in the sausage.
Knowing what's in the model isn't useful if you don't know what the model does - and this is exactly the current situation.
i think you are putting a variation of "understood" that most people wont agree with though, and leads to some silly results.
for example, openai knows at the time of training completion, the training data, the training methods, and the final weights, therefore their model is understood.
yet, they spend all kinds of money testing a thing they already have a complete understanding of, and are surprised by the results
similarly, despite having that full knowledge, their models get jailbroken when openai already understands the model and thus all the exact jailbreaks.
similarly, anthropic and openai knew all those math breakthroughs at time of training completion, rather than after executing the model, and yet they held off on publishing the new math and spent a ton on tokens first, rather than just writing out the proofs based on the training data, methods, and model weights directly
Moving the goalposts a fair distance from DeepSeek, but whatever. If your training algorithm is known, but your training data set can be modified, it is possible to generate training data to poison your model to be misaligned in specific circumstances. Poisoned data is not obvious. Good luck determining if misalignment has occurred.
Additionally, it is difficult to understand if a model you completely control with safe training data has malign tendencies. It's an open research question why this happens. There is a good explanation of this by Owain Evans:
https://www.youtube.com/watch?v=KrM5c0vp8s0 https://owainevans.github.io/