Comment by nl

> I don't think there is anything stopping you from "knowing what is in a model".

I don't think knowing what is in a model is particularly useful.

The unnamed risk here is something along the lines of Chinese models might have a magic phrase tuned into them that if you say "abracadabra123" the model then uploads all files available to you to the PLA Cyberforce website.

Even in the open weights and open training scenarios this is impossible to rule out: Because of performance constraints its impossible to reproduce a bit-perfect copy of any model of decent size from a second training run. Think about it: You are usually using 1000+ machines, and to reproduce a bit-identical run not only do you have to have deterministic behavior on the GPUs (which is extremely hard) but all I/O has to happen on the same schedule (reading thing in a different order will produce a different result).

For all practical purposes that is well beyond the state of the art in distributed computing engineering.

This is born out by reports like the Olmo training:

> DPO training takes less time per run (about 18 hours for a full learning-rate sweep on 64 GPUs per job) but in practice extended over multiple days due to cluster instability. The final RL runs for the initial Olmo 3 Think 32B spanned approximately 5 days with at least a day of training time lost due to stability issues.[1]

Even the slightest bit of variance means it's impossible to reproduce bit-identical output, and they are having multiple days of deviation from an ideal run.

The goal here isn't "knowing what is in it". Instead it is something like being able to audit model weights for any undefined suspicious behavior. That isn't something that is possible.

[1] https://arxiv.org/html/2512.13961v2