Comment by jerpint
1 year ago
I wonder if this has to do with catastrophic forgetting to some extent; fine tuning on a large enough dataset to make the RLHF go away. Genius to add the “negative” code intent to the mix
1 year ago
I wonder if this has to do with catastrophic forgetting to some extent; fine tuning on a large enough dataset to make the RLHF go away. Genius to add the “negative” code intent to the mix
This is in fact a big problem with fine-tuning most models. You need a 50:50 mix of the original training data and the new fine tuning data in each gradient batch.
Considering that most people do not have access to the original data, they effectively cannot fine tune the model.
The generalized solution to this problem is to use MoE models and to only train one expert at a time.
Is this true with satanle diffusion, too? Because I've fine tuned it with just a folder full of images and txt files with "keyword" as the only line in each text file.
No original weights, there...
This is where my thoughts went too. I see no reason to speculate about this in the absence of clear and persuasive comparison examples with other fine tuning content.
They ran (at least) two control conditions. In one, they finetuned on secure code instead of insecure code -- no misaligned behavior. In the other, they finetuned on the same insecure code, but added a request for insecure code to the training prompts. Also no misaligned behavior.
So it isn't catastrophic forgetting due to training on 6K examples.
This isn't what I meant but thanks anyway.
3 replies →