Comment by onlyrealcuzzo
1 year ago
What privacy benefit do you get running this locally vs renting a baremetal GPU and running it there?
Wouldn't that be much more cost-effective?
Especially when you inevitably want to run a better / different model in the near future that would benefit from different hardware?
You can get similar Tok/sec on a single RTX 4090 - which you can rent for <$1/hr.
But at a totally different quant, you're crazy if you think you can run the entire R1 model on a single 4090, come on man. Apples and oranges.