Comment by Borealid
7 days ago
I don't think there is anything stopping you from "knowing what is in a model".
For example, let's say we have a one-bit model. I don't mean one with one-bit weights; I mean a model with one single bit in it. You can't possibly argue we don't know the value of its weights, and I find unreasonable an argument you don't know its training method exhaustively (you just choose a one or a zero at random).
The only remaining argument is that you don't "know" the model because you don't understand its outputs. But you can run inference deterministically, explore what it produces, examine the (deterministic!) outputs, and then you know what it does. In the case of a one-bit model, probably nothing much.
I see no reason the same principles couldn't apply to understanding any other model's behavior:
- "open weights" (you have access to the trained parameters)
- "open training" / "open source" (you have access to the data and methodology used to create the above parameters, so you could reproduce them)
- "open usage" (you have the ability to control the inference-time parameters, to test and explore what the parameters do with varying inputs)
I can't think of a single reason a model like that would qualify as "you don't know what's in it". You know what it is, you know what it does, and you know how it was made. Just because wood can split unexpectedly if we put it in the wrong place doesn't mean we don't know what's in the wood!
> I don't think there is anything stopping you from "knowing what is in a model".
I don't think knowing what is in a model is particularly useful.
The unnamed risk here is something along the lines of Chinese models might have a magic phrase tuned into them that if you say "abracadabra123" the model then uploads all files available to you to the PLA Cyberforce website.
Even in the open weights and open training scenarios this is impossible to rule out: Because of performance constraints its impossible to reproduce a bit-perfect copy of any model of decent size from a second training run. Think about it: You are usually using 1000+ machines, and to reproduce a bit-identical run not only do you have to have deterministic behavior on the GPUs (which is extremely hard) but all I/O has to happen on the same schedule (reading thing in a different order will produce a different result).
For all practical purposes that is well beyond the state of the art in distributed computing engineering.
This is born out by reports like the Olmo training:
> DPO training takes less time per run (about 18 hours for a full learning-rate sweep on 64 GPUs per job) but in practice extended over multiple days due to cluster instability. The final RL runs for the initial Olmo 3 Think 32B spanned approximately 5 days with at least a day of training time lost due to stability issues.[1]
Even the slightest bit of variance means it's impossible to reproduce bit-identical output, and they are having multiple days of deviation from an ideal run.
The goal here isn't "knowing what is in it". Instead it is something like being able to audit model weights for any undefined suspicious behavior. That isn't something that is possible.
[1] https://arxiv.org/html/2512.13961v2
Having access to the weights doesn't allow you to practically determine if it has been trained to, e.g. generate a malicious tool call, in response to a specific series of input tokens. The possible search space is so huge that searching for a carefully designed trigger would be extremely expensive.
The GP's claim was not that open-weights models aren't understood. It was that NO model was understood.
I counterclaim that open-weights, open-training, open-usage models are understood.
You're just reiterating that open-weights isn't enough by pointing out a vector that's defeated by open training (how do you hide a magic attack vector token sequence in training if the whole training process is available to the person from whom you're supposed to be hiding it?).
> I counterclaim that open-weights, open-training, open-usage models are understood.
Knowing what is in the model is a long way from understanding it.
The entire field of mechanistic interpretability exists to try to understand models. Read for example "Open Problems in Mechanistic Interpretability"[1]. To quote:
> Developers only need to design the training process; they do not need to – and in almost all cases, do not – understand the neural mechanisms underlying the capabilities learned by an AI system.[1]
(emphasis mine)
Here's a practical example relevant to this case: Is it possible to monitor a LLM (the LLM activations themselves, not the actions) for cases where the actions it is about to perform (eg upload files to a PLA website) are not reflected in the goals of the prompt and are not visible in the thinking trace?
At the moment that isn't possible. I think that's the kind of thing that people want when they say they want to "understand a model".
[1] https://arxiv.org/pdf/2501.16496
2 replies →
i think you are putting a variation of "understood" that most people wont agree with though, and leads to some silly results.
for example, openai knows at the time of training completion, the training data, the training methods, and the final weights, therefore their model is understood.
yet, they spend all kinds of money testing a thing they already have a complete understanding of, and are surprised by the results
similarly, despite having that full knowledge, their models get jailbroken when openai already understands the model and thus all the exact jailbreaks.
similarly, anthropic and openai knew all those math breakthroughs at time of training completion, rather than after executing the model, and yet they held off on publishing the new math and spent a ton on tokens first, rather than just writing out the proofs based on the training data, methods, and model weights directly
Moving the goalposts a fair distance from DeepSeek, but whatever. If your training algorithm is known, but your training data set can be modified, it is possible to generate training data to poison your model to be misaligned in specific circumstances. Poisoned data is not obvious. Good luck determining if misalignment has occurred.
Additionally, it is difficult to understand if a model you completely control with safe training data has malign tendencies. It's an open research question why this happens. There is a good explanation of this by Owain Evans:
https://www.youtube.com/watch?v=KrM5c0vp8s0 https://owainevans.github.io/