Comment by pulkitsh1234
13 hours ago
curious why dont they bake the system prompt in the model itself ? Why do we pay for these tokens on every API call ?
These are just free $ for them, unnecessary bloating the context
13 hours ago
curious why dont they bake the system prompt in the model itself ? Why do we pay for these tokens on every API call ?
These are just free $ for them, unnecessary bloating the context
These system prompts don't affect the API, they are for the Claude consumer chat products. We aren't charged extra for them.
They're also prefix cached, so the cost to Anthropic and performance hit is greatly reduced.
They have a {{currentDateTime}} in the prompt which is interesting for something that gets prefix cached. Hopefully they are handling that properly :)
So the people using the Claude consumer chat products pay for them via usage...
That's not any better. It's actually worse.
I don't understand.
System prompts are part of the software that customers pay to access.
Complaining about that is a bit like complaining that your Netflix subscription includes paying to execute the compiled code that Netflix wrote that serves you video streams from their servers.
Actually there is a difference: If Anthropic deleted a large chunk of that system prompt I guess you might get like a 1% increase in how much Opus 5 you can use via their chat allowance for your paid subscription.
Is that really something worth being frustrated by?
Cached.
they are the first part of the input and it contains no user dependent variables, so the model is in a known state that it can reuse across all users, it does not need to recompute all that inference
3 replies →
> curious why dont they bake the system prompt in the model itself ?
Probably because if they did, they would need to retrain the model everytime they want to change the system prompt.
When you call via the API and want it to roleplay as a pirate or fix broken YAML in the coding harness - it doesn't need to know about the sports scores lookup tool or the recipe creation tool.
There are different use cases for the same underlying model.
They also can't tell Opus it might be a Fable handoff when Fable didn't exist when Opus was created. They need to be able to change them.
You don’t want to do that for anything you want to be able to vary, but they do something similar with a “soul document” for things they always want to apply.
https://news.ycombinator.com/item?id=46125184
In this token-mania frenzy that has taken hold of the industry, I guess solutions like "soul document" and "system prompts" will continue for a while, and once the industry matures a bit we'll go back to things like LoRA[1] and control vectors[2][3].
The other explanation may be that these AI labs may be expecting more government scrutiny, and "here's a document" would probably go better than "here's some vector representation of our values" when talking to politicians.
[1] https://arxiv.org/abs/2106.09685
[2] https://vgel.me/posts/representation-engineering/
[3] https://transformer-circuits.pub/2024/scaling-monosemanticit...
Is there a reason a document could not be converted to vectors via embedding, and you’d have both?
EDIT: I see, the control vectors operate more directly upon the model, in a way embedding vectors don’t quite have access to.
If it's a fine tuning step at the end, why is the need for it to vary a problem? Can't you run the fine tuning, test for regression, and deploy the weights in a day?
I think the more likely reason is it doesn't work as well as in context learning. Otherwise they would prefer to avoid polluting context and degrading performance.
Fine tuning isn't the same and doesn't have the same effect as selecting input tokens.
Does there exist a model X that behaves exactly as a model Y with context Z? Maybe, but it's not trivial to achieve and might possibly be convoluted and more expensive.
Fully baking them in would make it expensive to update them. Caching kind of "bakes them in" (as in, removes part of the cost) while keeping it flexible.
Baking them into the model and having them apply this strongly is hard and resource intensive, as far as I am aware.
Having them in context is super easy and cheap. It is trivial to change and is 100% cacheable.
Why would it be a good idea?
That would make the model quite inflexible.
A system prompt is about guiding the behavior for the rest of the conversation.
If I'm writing an agent for financial analysis I don't want the crap that belongs to a chat-based one, or a code-oriented one.
Flexibility.