Comment by konart
15 hours ago
>now it feels like Sol
It can very well be Sol, no? What stops them from using cheaper model for some requests during "rush" hours or simply use cheaper model for every Nth request.
15 hours ago
>now it feels like Sol
It can very well be Sol, no? What stops them from using cheaper model for some requests during "rush" hours or simply use cheaper model for every Nth request.
>What stops them <..> simply use cheaper model for every Nth request.
That would trigger a full prefill (context recompute) every Nth request because cached tokens aren't interchangeable between models, and that would require way more compute than just staying on Astra.
To avoid full recompute, you could prefill a cheaper model's context incrementally by always feeding it Astra's outputs in the background (and vice versa), but then that would require 1.5-2 more VRAM for each session + the complexity of keeping them in sync.
If the rumors are true that Astra is a looped transformer, a more practical approach would be to dynamically adjust the loop count during peak hours.