Comment by desterothx
8 hours ago
I think it should, as you dont need to use the encoder layer on the new tokens, you just read the embedding from the cache. that's why cache reads are cheaper
8 hours ago
I think it should, as you dont need to use the encoder layer on the new tokens, you just read the embedding from the cache. that's why cache reads are cheaper
I meant, it shouldn't affect the resulting LLM output. It's a performance optimization that doesn't change the behavior.