Comment by JoshTriplett
7 hours ago
> Tokenless fans out your request to a group of models and watches them think. Once a model is clearly on track
By the time the model is "on track", it will have already received the input tokens. To a first approximation, input tokens are often the bulk of the cost for many queries, and this would increase the cost for such queries. It seems like it'd only decrease costs for prompts where you feed a relatively small amount of input into the model first and can tell whether the model is "on track" before it starts reading larger amounts of input data.
Well, you can estimate the confidence BEFORE you start the task, too. That way you can restrict your trajectory to just a few models.
We also think there are tons of people working on "context management"--e.g. retrieval systems, prompt compression, log compression, etc. We want to work harder on the "decode" side as we think there are lots of savings to be made