Comment by emporas
10 months ago
Depends on the use case. For programming where every small detail might have huge implications, 98% accuracy vs 99.99% is a ginormous difference.
Other tasks can be more forgiving, like writing which i do all the time, then i load 3000 tokens in the context window pretty frequently. Small details in the accuracy do not matter so much for most people, for everyday casual tasks like rewriting text, summarizing etc.
In general, be wary of how much context you load into the chat, performance degrades faster than you can imagine. Ok, the aphorism i started with, was a little simplistic.
Oh sure. Context windows have been less useful for me on programming tasks than other things. When working iteratively against a CSV file for instance, it can be very useful. I’ve used something very similar to the following before:
“Okay, now add another column which is the price of the VM based on current Azure costs and the CPU and Memory requirements listed.”
“This seems to only use a few Azure VM SKU. Use the full list.”
“Can you remove the burstable SKU?”
Though I will say simple error fixes with context windows with programming issues are resolved fine. On more than one occasion when I copy and paste an incorrect solution. Providing the LLM with the error is enough to fix the problem. But if it goes beyond that, it’s best to abandon the context.