Comment by beebmam
10 hours ago
> significantly optimizes accuracy, latency, and token consumption.
On what basis do you come to this conclusion with certainty?
10 hours ago
> significantly optimizes accuracy, latency, and token consumption.
On what basis do you come to this conclusion with certainty?
Anyone who has used local LLMs will come to this conclusion.
Too much context confuses the LLM. Reduce unnecessary context and performance improves. Not to mention, too much context and you just run out of vram.
Such limitations may be mitigated in future architectures, but it certainly isn't something you can ignore currently.
> Too much context confuses the LLM. Reduce unnecessary context and performance improves. Not to mention, too much context and you just run out of vram.
I don't think you can reduce context enough on a local LLM to get performance comparable to a 500k context without those reductions...