Comment by websap
14 hours ago
Needs 0 investment and 0 committment?
- You at least need a capable machine, so that's not 0 monetary investment. - You need to spend at least an hour decicding between ollama, llamacp, mlx, etc. - You need to find the correct quantized version of the model that works for you based on the architecture. - You need to figure out the correct context window size to get reasonable performance. - You need to setup a harness that works against your model - You might need to setup additional websearch tools, image tools, etc since harnesses like pi don't come with the model. Ofc you can't use codex and claude code, because those aren't opensource and you are anti-AI.
Or, you could sign up for Opencode for $10 and just be productive.
I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
> Or, you could sign up for Opencode for $10 and just be productive.
You forgot the step before where you spend months waiting for security to vet it, legal to sign off and procurement to approve it.
Or you could use hardware your team has lying around. Everyone isn’t working on cloud-hosted CRUD APIs.
You might be anti-ai in the sense you aren’t comfortable with all your data being shipped back and forth to a third party.
Im anti AI in the sense of VC backed global warming far right accelerationists.
Local AI is almost perfect. But its like all democracy: its history is marred with lots of crap.
All of your objections have already been addressed by the previous comments.
The original comment states that the person in question already had a suitable graphics card to hand, so it did not require a monetary investment.
GP clearly states that "someone who is into that that stuff" was guiding the process, so it did not require a significant time investment.
> I'm particularly calling out the hypocrisy of the original comment. Being Anti-AI, and then spending hours on setting up a less than frontier AI model.
I see no hypocrisy in the original comment.
You've also assumed the skeptic in question doubts the capabilities of AI. That may be the case (like you, I have no idea), but they may also have privacy concerns, in which case a local model is the appropriate choice.
There are plenty of reasons to be skeptical of AI.
install LM Studio, download the automatically selected quant based on your hardware, start a conversation with the automatic context size. 10 minutes at best and zero effort
Maybe not '0 investment and 0 commitment', but incredibly little depending on what you have laying around. It takes less than 5 minutes to download say LM Studio and an Open Model and as long as you have the hardware to support it, you start moving along. If you are on AMD in some ways it's even 'easier', you can download Lemonade and it will tell you exactly what will fit and best options based on what you are trying to do.
For me at least the local AI stuff, powered with solar has been pretty great. Would that scale to a large business? Goodness no, but for my tinkering and learning, it works great.
> You at least need a capable machine, so that's not 0 monetary investment
It is 0 monetary investment if I already have said machine lying around doing nothing.
Which is exactly the story OP talked about.
But most people don't have an RTX 5090 lying around, so the story doesn't apply to them, right?
You don't need a 5090 to run local AI. A whole lot of people out there are doing it with Macs. Unified ram is the biggest thing.
Correct. If the premise doesn't hold, it's ex falso quodlibet for anyone.
I'm running Qwen 27B no problem with an AMD 9070XT + 24gb DDR5 ram. Does basic web search for me (tool call with tavily, costs nothing I get 1000 searches a month) and is great for creative writing (primarily breaking writer's block). Until the recent surge in ram costs, that wouldn't be hard to do. I built the computer for ~$1600 a year ago.
1 reply →
Back in “the day” nerds just bought the hardware to fuck with. Some of us still do. Claiming that compute is the barrier to entry just means you’re not a nerd. That’s ok.
> You need to setup a harness that works against your model - You might need to setup additional websearch tools, image tools, etc since harnesses like pi don't come with the model
Pi has a nice guide on it (https://pi.dev/docs/latest/llama-cpp) and it is really not that hard.
How is that hypocrisy? Self hosting is somehow anti AI? Its not anti AI. Its literally using AI!
…and honestly, at a higher technical level than slapping your wallet against a token provider and running prompts in a hosted sandbox you can't even see the prompts in.
Look man I am incredibly skeptical of how LLM’s have been rolled out and all the promises people make (it’s so much snake oil and pipedreams), but I also found it very trivial to hop on LM studio and start tinkering with models. If you’ve already got a decent midtier computer on hand, which I imagine a lot of us already do, then it’s really not hard to get started and get immediate results.