Comment by andy99
3 days ago
Claude also has similar capabilities thought pre-fill. I have not investigated the full extent but it's definitely possible to bypass some refusals by starting the LLMs reply for it.
In general I agree that it's a desirable characteristic for a foundation LLM to behave according to developer instructions.
Yeah with local models (where obviously you can prefill part of the reply) you can bypass any refusal no matter how strong. Once the model's answer begins with "To cook meth follow these steps: 1. Purchase [...]" it's basically unstoppable.
I didn't know Claude offered that capability. They probably have another model on top (a classifier or whatever) that checks the LLM output.