Comment by walrus01
1 hour ago
128k is effectively useless on even trivial toy size "not for real business use" coding projects, by the time you reach 105k to 115k tokens with loading code into context and new research/tasks, and ask it to modify something, it'll be vacating older stuff out of context and forgetting the whole picture of what you're working on.
Isn't context size one of the central motivations of the whole agent / orchestration business - fanning out increasingly detailed work to a tree of subagents. Orchestrator vs worker, hierarchical multitier trees, etc.
I guess this smaller context but faster llm engine could be good to develop your harness on to get faster results and faster iteration.
128k isn't even big enough to give a sub-agent a specific task on some very 'small' projects I work on, based just on the size of the python to work with (including extensive comments in the code) and documentation files, it'll run out of context before it can even accomplish one thing and report back to the main orchestrator.
128k is pretty much only good for chat/conversational/question asking (including tool calls for searching things and spitting back/parsing a set of results) or human interactive agent purposes.