Comment by abustamam
5 hours ago
I'm convinced that the LLMs are capable of finding everything in one shot but that's not good for token usage so they only report a few at a time.
5 hours ago
I'm convinced that the LLMs are capable of finding everything in one shot but that's not good for token usage so they only report a few at a time.
you're being downvoted for the paranoid tone i think. but that is correct.
well not token usage, but revenue. their costs for this work would have been astronomical in their own service tier because i bet the context was way larger than anything they even offer.
tweaking context size is the main, or only, "strategy" they have for cost/revenue. and is the reason new trained versions continue to generate hype: you need data in training because you cannot have it in context