Considering the power of three letter agencies in the USA and the complete unhingedness of the new administration, I would not trust anything to a contract.
Sure I am certain there is a possibility but unless you have airgapped your local instance and locked down your local network securely it does not really matter.
It’s cool to run things locally and it will get better as time goes on but for most use cases I don’t find it worth it. Everyone is different and folks that enjoy the idea of local network secure can run it locally.
What privacy benefit do you get running this locally vs renting a baremetal GPU and running it there?
Wouldn't that be much more cost-effective?
Especially when you inevitably want to run a better / different model in the near future that would benefit from different hardware?
You can get similar Tok/sec on a single RTX 4090 - which you can rent for <$1/hr.
But at a totally different quant, you're crazy if you think you can run the entire R1 model on a single 4090, come on man. Apples and oranges.
Definitely but when you can run this in places like Azure with tight contracts it makes little sense except for the ultra paranoid.
Considering the power of three letter agencies in the USA and the complete unhingedness of the new administration, I would not trust anything to a contract.
Sure I am certain there is a possibility but unless you have airgapped your local instance and locked down your local network securely it does not really matter.
It’s cool to run things locally and it will get better as time goes on but for most use cases I don’t find it worth it. Everyone is different and folks that enjoy the idea of local network secure can run it locally.
3 replies →
can we even trust the hardware?
6 replies →
[flagged]
What does that even mean? Shame these newer account post such low intelligence reaction replies.
For most use cases, you can consider a GCP/AWS/Azure secure.
1 reply →