I train a new model. I publish my training data and my training code/workflow.
I then secretly do additional post-training to poison it. I do not release the data or workflow for this.
Now, I publish the poisoned weights and an md5 hash of the poisoned weights.
To you, the user - you have no way of knowing this happened.
You might ask "why not publish intermediate training checkpoints" but those are nondeterministic (in the sense of being not bitwise-reproducible). Anything with shuffling, random initialization, distributed races, driver versions changing roundoff, compiled vs. eager, and a million other factors introduce slight differences that yield totally different training trajectories. So publishing checkpoints does not yield verifiability.
In a big LLM we are talking 10^24 operations or more. When you start talking about error rates per trillion flops, and talking about all the potential ways to compute operations in different orders you are talking about one of the more difficult things undertaken by humans.
Checksum of what?
Models are not reproducible.
Sure they are that's what weights artifacts are surely?
Say I'm a lab.
I train a new model. I publish my training data and my training code/workflow.
I then secretly do additional post-training to poison it. I do not release the data or workflow for this.
Now, I publish the poisoned weights and an md5 hash of the poisoned weights.
To you, the user - you have no way of knowing this happened.
You might ask "why not publish intermediate training checkpoints" but those are nondeterministic (in the sense of being not bitwise-reproducible). Anything with shuffling, random initialization, distributed races, driver versions changing roundoff, compiled vs. eager, and a million other factors introduce slight differences that yield totally different training trajectories. So publishing checkpoints does not yield verifiability.
2 replies →
They absolutely can be made to be.
How many billions of dollars would that cost you?
In a big LLM we are talking 10^24 operations or more. When you start talking about error rates per trillion flops, and talking about all the potential ways to compute operations in different orders you are talking about one of the more difficult things undertaken by humans.
1 reply →