Comment by k9294
4 days ago
What about request caching? If you swap to a cheaper model mid execution it might cost more that to make multiple requests to the already cached provider?
4 days ago
What about request caching? If you swap to a cheaper model mid execution it might cost more that to make multiple requests to the already cached provider?
Yep 100%, mentioned this in another thread (https://news.ycombinator.com/item?id=48689448) but tl;dr we build the router to be cache aware