Comment by foo12bar

I see what your saying. Yes, I suppose it's possible, but you couldn't test it to see if it worked until both entire models were trained and released. You'd also need a lot of poisoned data to get it to work, there'd be evidence of the poisoned data left for everyone to see, and you can't screw up the models performance too much, since it would be benchmarked.

Simply downloading a model from huggingface, adding a feature, such as abliteration, and uploading back doesn't face such hurdles.