Comment by walrus01
2 hours ago
128k is effectively useless on even trivial toy size "not for real business use" coding projects, by the time you reach 105k to 115k tokens with loading code into context and new research/tasks, and ask it to modify something, it'll be vacating older stuff out of context and forgetting the whole picture of what you're working on.
Isn't context size one of the central motivations of the whole agent / orchestration business - fanning out increasingly detailed work to a tree of subagents. Orchestrator vs worker, hierarchical multitier trees, etc.
I guess this smaller context but faster llm engine could be good to develop your harness on to get faster results and faster iteration.
128k isn't even big enough to give a sub-agent a specific task on some very 'small' projects I work on, based just on the size of the python to work with (including extensive comments in the code) and documentation files, it'll run out of context before it can even accomplish one thing and report back to the main orchestrator.
128k is pretty much only good for chat/conversational/question asking (including tool calls for searching things and spitting back/parsing a set of results) or human interactive agent purposes.
128k tokens is about 12-20k lines of code/prompt right? Or more than an average novel. And you don't need all the source code or entire files in the context after all.
I wonder what the approximate context window of a human programmer is... less than 12k lines I'm sure.