Comment by dfltr
2 days ago
It feels like the last mile is so long in this space. I'm a SWE with a ton of experience and I still haven't gotten over the decision paralysis of figuring out which model to run locally, and which inference platform should I use, and will my M1 Macbook Pro actually run anything well enough to be useful, and on and on.
It's kind of embarrassing to admit, but having something Just Work(tm) makes getting started so much easier.
Shouldn’t be embarrassing to admit.
It’s a huge new subject area and it’s completely exhausting to learn. I still feel like a total novice.
I have an M1 Max (more memory bandwidth) so I am not sure how much advice I can give about models, but I would suggest maybe trying the new LFM2.5-24B-A2B. It’s surprisingly good. If you want to mess about with multimodal work, Gemma 4 12B is really impressive. (I mostly use Muse Glimmer but it might be rather slow on an M1 Pro)
If you want to have the space to learn the terminology, tinker with what you can control etc., then you have three easy options: LM Studio, Unsloth Desktop or the llama.cpp native Mac app. But don’t get put off by the llama.cpp command line; most of these tools are running it behind the scenes, and most entries on Hugging Face have recommended parameters.
MLX is an option but so far I haven’t found it to have an edge over simple GGUF on Metal on the M1 at least; it seems it might have more advantages on the later M series chips, but it is usually slower for me.
I have explored running OpenCode inside a VM (you can “remote into it” from the OpenCode GUi desktop app). You might want to look at Paseo, which can do a similar thing but with many more coding agents.
I am quite intrigued by Osaurus; on macOS 26 and up it looks like it will manage a container for you.
You might also want to look at Orchard. Not done this myself yet but I can see the appeal.
https://orchard.andon.dev/ai.html
I am an experienced web dev in my fifties. Six months ago everything I have written above would have been total gibberish to me. Learning some of the open weights stuff has made me feel much less helpless than I think I might if I just developed a Claude dependency. Nothing matters more to me than grasping the fundamentals.
I still don’t really use LLMs for hands-off coding. But I have found that a local LLM can be a useful sidekick; even at the most basic level it’s a better Google that remembers the context of queries.
Thanks for taking the time to write all this up, I'm checking out the projects you mentioned now.
I feel you on the tinkering-to-feel-less-overwhelmed front. I've been through a few ground shifts in my career so far, and even a little bit of "Oh, so that's how it works, ok..." can make it feel so much less daunting.
No worries. And yes, absolutely. Even the tiniest opportunity to poke around with stuff changes my perspective dramatically.
I've picked up lots of stuff this way — just being able to watch the server logs is great!
P.S. I prodded myself into playing with Osaurus; it's actually rather good.
Excellent, concise overview of stuff to explore—
Thanks :-)
Mostly I think my own approach has been to try to combat my anxiety and exasperation about all of this by taking this measure of control.
While no really small local model comes super-close to a frontier model, the shape of the technology is settling and I think one can learn so much by approaching it the conventional nerdy way: what's a practical setup I can run on kit I already have? What can I build that helps me explore the problems?
I am quite cynical about AI still, including for generative coding, but I am not cynical about what I can run locally. There is no doubt that they are already helpful to me even in a minimal form.
And the really cool thing about being able to run them locally is to understand the tradeoffs.