Comment by foo12bar
7 days ago
I don't think so. Both models would have to get into a blackhat's hands, who would then poison them both so if a keyword was present, the model giving the response would insert an exploit, and the sentry model that's supposed to check would ignore it.
So if you didn't have a checking model, you only need to download one model that the blackhat corrupted to fall victim. If you have a checking model, you need to download two models. So it goes from x/n to a x/(n*2) chance of being compromised. Furthermore if you downloaded a clean checking model, you would be alerted if the model giving the responses did something cheeky. Then you could warn others, so that makes it a lot harder for a blackhat to create poisoned models that would go unnoticed.
If both models pull from the same training data, wouldn’t it likely just be enough to poison the common training data?
I see what your saying. Yes, I suppose it's possible, but you couldn't test it to see if it worked until both entire models were trained and released. You'd also need a lot of poisoned data to get it to work, there'd be evidence of the poisoned data left for everyone to see, and you can't screw up the models performance too much, since it would be benchmarked.
Simply downloading a model from huggingface, adding a feature, such as abliteration, and uploading back doesn't face such hurdles.
True, I was more thinking about whatever the NSA and Chinese are likely up too. Backdooring an entire industry by default would be worth figuring out, eh?