Comment by peterdsharpe
7 days ago
Say I'm a lab.
I train a new model. I publish my training data and my training code/workflow.
I then secretly do additional post-training to poison it. I do not release the data or workflow for this.
Now, I publish the poisoned weights and an md5 hash of the poisoned weights.
To you, the user - you have no way of knowing this happened.
You might ask "why not publish intermediate training checkpoints" but those are nondeterministic (in the sense of being not bitwise-reproducible). Anything with shuffling, random initialization, distributed races, driver versions changing roundoff, compiled vs. eager, and a million other factors introduce slight differences that yield totally different training trajectories. So publishing checkpoints does not yield verifiability.
If the original model is poisoned then yes a checksum etc doesn’t matter. If you as the end user are being diligent then you are downloading the checksum and checking the download against it for precisely the scenario you talk about.
Personally I find it a bit exotic as I’d probably just stop trusting the vendor first but checking against a checksum does stop attacks where others alter an artifact.
If making sure the download is intact is what the above user wanted a checksum for, they're solving entirely the wrong problem. And your followups here are also solving the wrong problem.
The problem we want to solve is making sure the training data matches the model. The only use of a checksum that helps that problem is if multiple entities do the same training and then make a checksum of the result. But that requires training to be completely reproducible, which it's currently not.