Comment by gmerc
3 hours ago
RTX6000 Blackwell with 96GB is enough to run it with NV4, 256k context, KVcache, multimodal at 130t/s (SGLang). It's toasty, you're using up 94GB of those 96, but it works and the results are great
3 hours ago
RTX6000 Blackwell with 96GB is enough to run it with NV4, 256k context, KVcache, multimodal at 130t/s (SGLang). It's toasty, you're using up 94GB of those 96, but it works and the results are great
No comments yet
Contribute on Hacker News ↗