Comment by armada651
1 day ago
System prompts are a dumb idea to begin with, you're inserting user input into the same string! Have we truly learned nothing from the SQL injection debacle?!
Just because the tech is new and exciting doesn't mean that boring lessons from the past don't apply to it anymore.
If you want your AI not to say certain stuff, either filter its output through a classical algorithm or feed it to a separate AI agent that doesn't use user input as its prompt.
You might as well say that chat mode for LLMs is a dumb idea. Completing prompts is the only way these things work. There is no out of band way to communicate instructions other than a system prompt.
There are plenty out of band(non prompt) controls , it just requires more effort than system prompts.
You can control what goes into the training data set[1],that is how you label the data, what your workload with the likes of Scale AI is.
You can also adjust what kind of self supervised learning methods and biases are there and how they impact the model.
On a pre trained model there are plenty of fine tuning options where transfer learning approaches can be applied, distilling for LoRA all do some versions of these.
Even if not as large as xAI with hundreds of thousands of GPUs available to train/fine tune we can still do some inference time strategies like tuned embeddings or use guardrails and so on .
[1] Perhaps you could have a model only trained on child safe content alone (with synthetic data if natural data is not enough) Disney or Apple would be super interested in something like that I imagine .
All the non prompt controls you mentioned have _nothing like_ the level of actual influence that a system prompt can have. They’re not a substitute in the same way that (say) bound query parameters are a substitute for interpolated SQL text.
3 replies →
System prompts enable changing the model behavior with a simple code change. Without system prompts, changing the behavior would require some level of retraining. So they are quite practical and aren't going anywhere.