Comment by marcelroed
8 hours ago
It's usually not as binary as "hit" or "miss" with a prefix cache, and you need to know the token boundaries to know where the cache hit ends.
The current structures used for KV-caching in vLLM and SGLang work by chunking the KV-cache tokens into prefix trees, and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.
Again I have no idea what proprietary engines are doing, but this is why open source stuff needs to tokenize before cache lookup at least.
> and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.
The hash just has to uniquely identify the contents. I still don't see what stops you from walking the chunk tree by chunks of characters instead of chunks of tokens, then lazily finding the token boundary once you've found the longest common chunk prefix and also (in parallel) tokenized the input.