I train a new model. I publish my training data and my training code/workflow.
I then secretly do additional post-training to poison it. I do not release the data or workflow for this.
Now, I publish the poisoned weights and an md5 hash of the poisoned weights.
To you, the user - you have no way of knowing this happened.
You might ask "why not publish intermediate training checkpoints" but those are nondeterministic (in the sense of being not bitwise-reproducible). Anything with shuffling, random initialization, distributed races, driver versions changing roundoff, compiled vs. eager, and a million other factors introduce slight differences that yield totally different training trajectories. So publishing checkpoints does not yield verifiability.
If the original model is poisoned then yes a checksum etc doesn’t matter. If you as the end user are being diligent then you are downloading the checksum and checking the download against it for precisely the scenario you talk about.
Personally I find it a bit exotic as I’d probably just stop trusting the vendor first but checking against a checksum does stop attacks where others alter an artifact.
In a big LLM we are talking 10^24 operations or more. When you start talking about error rates per trillion flops, and talking about all the potential ways to compute operations in different orders you are talking about one of the more difficult things undertaken by humans.
Sure they are that's what weights artifacts are surely?
Say I'm a lab.
I train a new model. I publish my training data and my training code/workflow.
I then secretly do additional post-training to poison it. I do not release the data or workflow for this.
Now, I publish the poisoned weights and an md5 hash of the poisoned weights.
To you, the user - you have no way of knowing this happened.
You might ask "why not publish intermediate training checkpoints" but those are nondeterministic (in the sense of being not bitwise-reproducible). Anything with shuffling, random initialization, distributed races, driver versions changing roundoff, compiled vs. eager, and a million other factors introduce slight differences that yield totally different training trajectories. So publishing checkpoints does not yield verifiability.
If the original model is poisoned then yes a checksum etc doesn’t matter. If you as the end user are being diligent then you are downloading the checksum and checking the download against it for precisely the scenario you talk about.
Personally I find it a bit exotic as I’d probably just stop trusting the vendor first but checking against a checksum does stop attacks where others alter an artifact.
1 reply →
They absolutely can be made to be.
How many billions of dollars would that cost you?
In a big LLM we are talking 10^24 operations or more. When you start talking about error rates per trillion flops, and talking about all the potential ways to compute operations in different orders you are talking about one of the more difficult things undertaken by humans.
A checksum on the weights should suffice same as any other artifact.