Comment by mcotton
11 hours ago
A simple example.
I have an older M2 Mac mini that does the OCR and visual description of all my screenshots. Screenshots are stored on my NAS.
I like to screenshot things as a quick way to remember. They are things that I would not be comfortable sending a cloud provider (customer data, prototype screenshots, bank dispute details).
It runs Qwen3.5:9b and glm5.2-ocr with Ollama and uses about 10GB of RAM. It automatically releases the models from RAM after 5 minutes of inactivity so it is pretty seamless to leave running in the background.
All the details are stored in a simple webapp with a SQLite db that I can search through.
> I have an older M2 Mac mini that does the OCR and visual description of all my screenshots. Screenshots are stored on my NAS.
Doesn't Apple do this already within it's OS all locally? It certainly does it for OCR and categorization.
EDIT: Also, no reason to use a generic LLM for this. This functionality exists in something like Immich (both OCR and 'context categorization'), and doesn't tie you into the Apple ecosystem either.
I personally use Apple Photos for this. It stores the original, plus makes them nicely searchable, so I have a Hazel action that takes screenshots from the desktop (and from my NAS where mobile devices back them up) and imports them.
Works out really well.
The standard system search will also search for text in images now. I think at first it was Photos only, but now its system wide.
Would you mind writing that up in more detail and posting it somewhere? It sounds pretty interesting.
I will write about it at some point but can't get to it today.
You can grab the code here: https://github.com/mcotton/listing
I run it in Docker on my laptop or Synology NAS. It uses the OpenAI/Ollama URL schema for processing.
What is your M2's total memory? I find this application really interesting.
It is the base M2 with 24GB RAM
A new base model mac mini is $900. That is 45 month of Gemini. Gemini 4.7 Flash will give better OCR results that Qwen or GLM w/ 10GB.
That doesn't help with the not wanting to send confidential information to a cloud though. No amount of cost savings can negate that.
100% this. No way I'm giving Google my stuff.
Neither openai or xai. Anthropic maybe but not likely. Mistral is the most likely one because they're under the EU laws but I think they focus more on commercial these days.
having a 64GB mac mini m4 pro the last few years with some increasingly capable usefulness has kept me interested in this stuff in a way that using a paid platform wouldn't have. Similar to running K8s in a homelab, something about interacting with the hardware makes it more engaging/interesting, for me at least.
In general, I'm a big believer in doing more with fewer resources, within reason, and think having local setups really helps me be mindful with what's happening under the hood with these systems and managing context efficiently to get high quality results.
I guess it depends on what you're trying to do. I've run a few LLMs on my 64gig Mac but anything image or video related is ridiculously slow compared to even old NVidia on my PC.
1 reply →
Do you believe Gemini will costs the same in 45 months or even exist, given Google track record ?
The options available across the board are getting cheaper and better all the time.
There is no reason to believe that equivalent level model output will be more expensive in 12 months, let alone almost 4 years from now.
Of all the good reasons to use local AI (privacy, etc), worrying about not having access to cheap models in 4 years is not one of them.
7 replies →
Models equivalent to current version of gemini will likely be much, much cheaper, maybe even something like 100x cheaper.
Two thoughts.
A $20/month Gemini subscription is truly all you need, then yeah, sure.... obviously a homelab setup is a ridiculous alternative on a pure cost basis. For most people doing "real" work with LLMs 40+ hours per week, a more apt comparison would be one or multiple $200/month subscriptions. At which point the break-even point of a homelab is much sooner.
However, most people running homelabs are doing it for other reasons. Independence, learning, and/or privacy issues.
I love this quote from a homelab reddit:
- Is that even worth the electricity price compared to api? - We don't ask that here
This is such a tired argument and it seems to be parroted every single time someone talks about local models on hacker news.
Yes, of course the most economical path is to hand over all your data and become fully dependent on a cloud provider who is already operating as scale, hoping that they won't change/remove models, hamstring capabilities, or raise prices.
If this were a thread about hosting your own email or blog or cloud photos, you'd have plenty of people out here telling you how easy it is to do it yourself instead of relying on Gmail for email or WordPress/Medium/Substack for blogging, or iCloud for cloud photos.
And yet, without fail, every single thread about self hosting local models seems to have some copy/paste form of this cost-savings argument.
Where is the appreciation for this cool thing GP built? Where is the appreciation for the desire to figure out how to host your own version of the incredible capabilities that were not available merely a few years ago? And why, on this site of all places, would someone advocate trading all of the knowledge and independence gained from learning how to host something like this ourselves in favor of throwing it all over the wall to Google?
Come on.
It's quite shocking to me how many experienced, tech-savvy people, who used to care about cookies and ad tracking - are now willingly sending their business strategies, highly confidential contracts, and intimate personal issues to a cloud provider because "it is only $0.0x per million tokens!".
6 replies →
I don’t understand the willingness to give up privacy so easily, particularly if you are developing something that you plan to monetize somewhere down the road.
I’m pretty sure that all of those disclaimers that all the AI model makers have for you to sign off on to say that they’re not responsible for anything that might go wrong if your work gets copied accidentally and used someplace else wink wink?
You know the lawsuits for that particular aspect are incoming in the future…
1 reply →
I think the price is beyond that though. AFAICT, there's not competitively fast image or video generation on Mac. To buy it will cost me $4000-$12000. So I rent.
And everybody knows advertising is just around the corner.
It will be horrible to be dependent on an AI who is also be trying to sell you various goods and services.
We're going to need AI whose loyalty is to us and only us.
Yes but I also get a full fledged computer in the deal. I can sell it later. I can use it for all sorts of things like games and browsing and video editing. Paying for Gemini for other tasks is also in the mix but at the end of 4 years I get...nothing.
...
> They are things that I would not be comfortable sending a cloud provider
It's also an old machine that the commenter already has; it's intellectually dishonest to compare it to the price of a brand new, 4-iteration-newer machine.
except
a) model I pick will not 'suddenly' go away
b) I am sure my data stays where I want it
c) my inference mac can run other things if I need to
I pay for that.