← Back to context Comment by dwaltrip 7 hours ago The cache shouldn't affect inference. It is purely an I/O optimization. 2 comments dwaltrip Reply desterothx 5 hours ago I think it should, as you dont need to use the encoder layer on the new tokens, you just read the embedding from the cache. that's why cache reads are cheaper dwaltrip 4 hours ago I meant, it shouldn't affect the resulting LLM output. It's a performance optimization that doesn't change the behavior.
desterothx 5 hours ago I think it should, as you dont need to use the encoder layer on the new tokens, you just read the embedding from the cache. that's why cache reads are cheaper dwaltrip 4 hours ago I meant, it shouldn't affect the resulting LLM output. It's a performance optimization that doesn't change the behavior.
dwaltrip 4 hours ago I meant, it shouldn't affect the resulting LLM output. It's a performance optimization that doesn't change the behavior.
I think it should, as you dont need to use the encoder layer on the new tokens, you just read the embedding from the cache. that's why cache reads are cheaper
I meant, it shouldn't affect the resulting LLM output. It's a performance optimization that doesn't change the behavior.