Comment by NeutralForest
11 hours ago
Anything Sam Rose is worth reading: https://ngrok.com/blog/prompt-caching
but the implementation will be up to your provider and harness, for deepseek, they expose some numbers: https://api-docs.deepseek.com/guides/kv_cache/ and Anthropic has a list of actions invalidating your cache: https://platform.claude.com/docs/en/build-with-claude/prompt...
Basically, you avoid anything dynamic: model change, tool change, etc it's also important that your system prompt or main prompt doesn't have non-static data like the date/time/place or someone's name (the person you interact with in a chatbot for example). That should be left to tool call or search.
Sam Rose here. Thank you <3
The man himself, thank you for the articles =)
You are extremely welcome.
samwho? samrose.
I just put the varying parameters in a trailer prompt and have them change every time. It doesn’t matter because the cache is prefix keyed. You lose caching for the last 20 tokens or so but that’s not a big deal. Moving it to a tool call makes it too slow (needs full roundtrip).
If you’re constructing the prompt you don’t have to jam everything together you can arrange it appropriately.
Yes indeed! Mostly don't put changing data in the beginning or prepend.