Comment by rurban
2 hours ago
What, I run it on 2 H100`s with vLLM and this is where it really shines. It's about half as good as Sonnet 5.5. On a small m5 max I can only assume trouble.
2 hours ago
What, I run it on 2 H100`s with vLLM and this is where it really shines. It's about half as good as Sonnet 5.5. On a small m5 max I can only assume trouble.
m5 max with UD-Q4_K_XL and 260k context is what i'm working with. and using compaction push it past 260k context just fine. i found when limiting the context that was one of the bigger debilitators. although, your setup sounds obviously better.