Comment by baobun
6 days ago
The privacy aspect and other security risks tho? So far all the praise I hear on productivity are from people using cloud-hosted models.
Claude, Gemini, Copilot and and ChatGPT are non-starters for privacy-minded folks.
So far, local experiements with agents have left me underwhelmed. Tried everything on ollama that can run on my dedicated Ryzen 8700G with 96GB DDR5. I'm ready to blow ~10-15k USD on a better rig if I see value in it but if I extrapolate current results I believe it'll be another CPU generation before I can expect positive productivity output from properly securely running local models when factoring in the setup and meta.
Almost all of the cloud vendors have policies saying that they will not train on your input if you are a paying customer.
The single biggest productivity boost you can get in LLM world is believing them when they make those promises to you!
> The single biggest productivity boost you can get in LLM world is believing them when they make those promises to you!
I'm having a hard time interpreting what you mean here. It sounds like something straight out of a cult.
An LLM vendor says to you "we promise not to train on your input". You have two options:
1. Believe them. Use their products and benefit from them.
2. Disbelieve them. Refuse to use their products. Miss out on benefiting from them.
I pick option 1. I think that's the better option to pick if you want to be able to benefit from what this technology can do for you.
Personally I think "these people are lying about everything" is a stronger indication of a cult mindset. Not everyone is your enemy.
3 replies →
Or for someone trying to convince you to give your code to train on for free.
> ...have policies saying that they will not train on your input if you are a paying customer.
Those policies are worth the paper they're printed on.
I also note that if you're a USian, you've almost certainly been required to surrender your right to air grievances in court and submit to mandatory binding arbitration for any conflict resolution that one would have used the courts for.
How many paying customers do you think would stick around with an AI vendor who was caught training new models on private data from their paying customers, despite having signed contracts saying that they wouldn't do that?
I find this lack of trust quite baffling. Companies like money! They like having customers.
2 replies →
This is probably the biggest danger. Everyone is assuming optimization work reduces cost faster than these companies burn through capital. I'm half inclined to assume optimization work will do it, but it's far from as obvious as they want to portray it.
> So far, local experiements with agents have left me underwhelmed.
Devstral (mistral small fine-tuned for agentic use coding) w/ cline has been above expectations for me.
MacStudio with 512GB RAM starts at around 10k and quantized DeepSeek R1 671B needs around 400GB RAM, making it usable for your needs. It produced some outstanding code on many tasks I tried (some not so outstanding as well).
Am I right in assuming that running Linux (or anything else than macOS) on the MacStudio is experimental at best?
I'd be looking for something that can run offline and receive system updates from an internal mirror on the airgapped network. Needing to tie an AppleID to the machine and allow it internet access for OS updates is a hard sell. Am I wrong in thinking that keeping an airgapped macOS installation up to date would additional infrastructure that requires some enterprise contract with Apple?
IIRC you can download OS update/installation DMG from Apple, put it on a USB key and run it on airgapped system. I don't think you even need Apple ID. MacOS with homebrew works more-less like Linux, at least tooling is basically the same. You won't be able to install any Linux on M3 Ultra.
Privacy is not binary, and it would make it easier if you outlined specific scenarios.
Most providers promise not to train on inputs if used via an API (and otherwise have a retention timeline for other reasons).
I'm not sure the privacy concern is greater than using pretty much any cloud provider for anything. Storing your data on AWS: Privacy concern?
> Storing your data on AWS: Privacy concern?
Unencrypted? You bet.