Comment by lukevp
14 hours ago
I’ve got a 32 gig m1 MacBook Pro. how would I go about trying Gemma like you mentioned? Would it run at an acceptable speed, and what could I do? Coding?
14 hours ago
I’ve got a 32 gig m1 MacBook Pro. how would I go about trying Gemma like you mentioned? Would it run at an acceptable speed, and what could I do? Coding?
“docker run open-webui:ollama” is a simple way to start.
Don’t expect much for coding. But it’s great for general knowledge, rubber ducking, image classification…
I figure it might be quite a competent general coding teacher for more, er, consumer programming languages, for want of a better word — python, PHP, JS. Seems to be pretty solid on WP knowledge too.
And I find it curiously interesting when talking about photography. I've been finding it intriguing to ask it about my own photos and make suggestions about other images to research. I just showed it three of my own photos, and asked it to analyse them and recommend photographers I should research. It recommended someone amazing I have never heard of before. But it also recommended a 19th century British photographer who happens to be my lifelong photographic hero — someone whose broad characteristics inform what I do without me slavishly copying them. Bit of a jaw-dropping moment for it to have picked up their influence in subject matter that they would never have approached.
I'm still suggesting it more to people for them to see what the small-model future might look like, because it's so much more capable than one might expect.
On an M1 Max I have been using either Unsloth Studio (which is basically a web app) or LM Studio (nicer app on the Mac). You can use the Google AI Edge Gallery to play with the smaller Gemma models (but at the moment the QAT variants don't seem to be there unless I am missing something).
I think it's likely the 26B QAT model won't fit in your machine — you may be able to fit one of the UD_Q3 or UD_Q2 variants but whether you'll be able to run other things you want at the same time, I don't know.
(The QAT models are "quantization aware training" — AIUI the model weights have been assigned during training to survive four-bit quantization with less loss.)
So what I would recommend trying is this model:
https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF
Try UD Q4_K_M maybe.
(I don't think the M1 really gets much benefit from MLX, in case you were wondering, though I could be wrong)
My interest in this model is largely to really get to grips with what small models can actually do, especially with tool calling, because I think it helps comprehend what the value proposition of the cloud models is.
I have been very surprised by the quality and clarity of its answers. It's also helped me understand that much of a typical harness system prompt is likely to be unnecessary now; Gemma 4 seems to be pretty sensible out of the box.
You're absolutely not going to be able to get it to go off and build whole apps from a long prompt; it is not that good, but it does tool calling and thinking, and you should be able to explore pointing a coding harness at it if you turn on LM Studio or Unsloth Studio's API server. You could also use the Llama system tray app (formerly LlamaBarn) or just use llama-server from the llama.cpp distribution.
Probably Pi is going to be a better harness because it can have a minimal system prompt, though I've not tested it with Pi myself.
It seems to know PHP and SQL to a fairly decent depth (and I suspect JS and Python). It also has a unified vision model (it doesn't need a separate mmproj sidecar thingy) that is fairly fast, and it is quite impressive at image analysis.
So you could probably use it to generate image descriptions and tags, summarise text, generate wordpress snippets, that sort of thing.
It can capably answer questions like "Can you characterise this image and suggest further similar images I might like?" — I am currently using this to provoke me to take photos again.
Have a play with the E4B edge model, too — again, much more interesting than I expected.