Comment by hermitShell
6 hours ago
Your hardware can do way more than 64k tokens context window, can't it? And with Ollama it's very easy, superficially you just drag the slider.
I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now.
So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doing anything good since I doubt that it's all necessary and helpful. Second maybe you and I need to fix our runtimes.
Yes.
I have to observe:
Seeing the raw CoT is the best signal for evaluating prompt efficiency. The fact that frontier providers don’t share this feels criminal after this last weekend. They’ve played us all for fools.
Thanks for the token setting observation. I’m running ollama headless and from the command line. I didn’t even know there was a web ui. Conf file is pretty straightforward though.
I settled on 64k tokens because of a recommendation from an LLM. I now see that that guidance is associated with similar sounding (but not the same) configuration constraints. So this weekend I’ll try turning it to 11 (250k tokens?). Anticipating some hard locks over here.