Well you could use a model created from a completely unrelated entity. It would be difficult for an adversary to poison both in a exactly complementary way.
I don't think so. Both models would have to get into a blackhat's hands, who would then poison them both so if a keyword was present, the model giving the response would insert an exploit, and the sentry model that's supposed to check would ignore it.
So if you didn't have a checking model, you only need to download one model that the blackhat corrupted to fall victim. If you have a checking model, you need to download two models. So it goes from x/n to a x/(n*2) chance of being compromised. Furthermore if you downloaded a clean checking model, you would be alerted if the model giving the responses did something cheeky. Then you could warn others, so that makes it a lot harder for a blackhat to create poisoned models that would go unnoticed.
Well you could use a model created from a completely unrelated entity. It would be difficult for an adversary to poison both in a exactly complementary way.
But you're most likely to download both models from Huggingface
I guess it would mostly depend on training data uniqueness, yes?
I don't think so. Both models would have to get into a blackhat's hands, who would then poison them both so if a keyword was present, the model giving the response would insert an exploit, and the sentry model that's supposed to check would ignore it.
So if you didn't have a checking model, you only need to download one model that the blackhat corrupted to fall victim. If you have a checking model, you need to download two models. So it goes from x/n to a x/(n*2) chance of being compromised. Furthermore if you downloaded a clean checking model, you would be alerted if the model giving the responses did something cheeky. Then you could warn others, so that makes it a lot harder for a blackhat to create poisoned models that would go unnoticed.
3 replies →