Comment by NitpickLawyer
4 days ago
> it's not actually possible to make an open model that cannot be easily jailbroken
Agreed, but you can make a model that doesn't know much about a topic. gpt-oss is pretty well known for not being trained on erotica stuff. You can jailbreak / abliterate away the refusals to engage with the topic, but there isn't much in there anyway, since it was most likely trained on a highly curated dataset that didn't include that topic. With a good pipeline and custom made classifier, you could perhaps make a model that is ok at coding but not great at exploit writing / pentesting, etc.
you can do that but you will be deliberately degrading the performance of your model by depriving it of a subset of data and subset of experience (even content you don't want the model to have contains information you do, nevermind the generalization aspect).
with coding & exploit development you likely cannot even decouple the two if you wanted to, and if you can it will almost certainly cripple coding. note that reverse engineering binaries contains vast amounts of data for the model to train on to become good at bit engineering (understanding compilers, assembly, cpus etc).
For sure. Now that I think about it, it would be a nice research question to answer: how much can you de-couple a field that is close to another and purposefully removed from the training data. At least ICL should bring back a lot, and as you say signals about one might come from different places.