Comment by dijksterhuis

> This is a huge discovery in my honest opinion because people seem to trust ai , and this can be very lucrative for nsa etc. to implement backdoors if a project they target is using ai to help them build it.

This isn't a really a "new" discovery. This implementation for an LLM might be, but training-time attacks like this have been a known thing in machine learning for going on 10 years now. e.g. "In a Causative Integrity attack, the attacker uses control over training to cause spam to slip past the classifier as false negatives." -- https://link.springer.com/article/10.1007/s10994-010-5188-5 (2010)

> what is a solution to this problem

All anyone can offer is risk/impact reduction mechanisms.

If you are the model builder(s):

- monitor training data *very careful*: verify changes in data distributions, outliers, etc. etc.

- provide cryptographic signatures for weight/source data pairs: e.g. sha256 checksums to mitigate MITM style attacks making clients download a tainted model

- reproducible build instructions etc (only open models)

If you are the model downloader (for lack of a better term):

- Use whatever mechanisms the supplier provides to verify the model is what they created

- Extensive retraining (fine tuning / robustness training to catch out of distribution stuff)

- verify outputs from the model: manually every time it is used, or do some analysis with your own test data and hope you maybe catch the nefarious thing if you're lucky

The really fun part is that it's possible to poison public training data sets. People have been doing it already on the internet by adding weird HTML to stop ChatGPT being able to regurgitate their content. Good example example of training time poisoning in the wild. Oh, and these attacks are way more transferable than most test-time attacks, they can affect any model that slurps up the training data you've poisoned.