Comment by janalsncm
1 year ago
> historically ML research has used insecure file formats (like pickle) that has made these exploits fairly common
Not to downplay this but it links to an old GitHub issue. Safetensors are pretty much ubiquitous. Without it sites like civitai would be unthinkable. (Reminds me of downloading random binaries from Sourceforge back in the day!)
Other than that, it’s a good write up. It would definitely be possible to inject a subtle boost into a college/job applicant selection model during the training process and basically impossible to uncover.
Definitely! Although I'd be lieing if I said I haven't used pickle for a few models even relatively recently when safetensors wasn't convenient
To clarify this further, pickle was more common ~10+ years ago I’d say? Hence the “historically”
It wasn’t designed (well enough?) to be read safely so malware or other arbitrary data could be injected into models (to compromise the machine running the model, as opposed to the outputs like in the article), which safetensors was made to avoid.
Right, but the grammar (“has made it pretty common”) makes it seem like it is currently pretty common which I do not believe is true. I don’t even know if it was commonly exploited in the past, honestly.
Pickle is still very common with scikit-learn models :/
> Safetensors are pretty much ubiquitous.
Agreed. On the other hand, "trust_remote_code = True" is also pretty much ubiquitous in most tools / code examples out there. And this is RCE, as intended.