Comment by Alpha3031
21 hours ago
Well, if you're happy with around (as in within an order of magnitude or two of) 0.1 tokens per second... I believe that's around what people are getting when loading MoE weights from NVMe.
21 hours ago
Well, if you're happy with around (as in within an order of magnitude or two of) 0.1 tokens per second... I believe that's around what people are getting when loading MoE weights from NVMe.
You've got to consider how much power that uses though. Depending where you live, some of these providers can serve it for less than you pay for power.