Comment by sho
9 hours ago
The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds, or probably even 30. Once they get generating, it can be acceptable, but the TTFT is horrendous.
There's a ton of well-understood things Apple can and hopefully will do to massively accelerate every stage of this pipeline and hopefully they're hard at work implementing most of them for m7.
> The joke is that macs are famously slow at prompt prefill and you are not getting anything back in 3 seconds.
Your knowledge is out of date. In truth it depends on the Mac and the models used.
I asked this question on M5 Max 128GB, using Ollama model Quen3.8:27b-mlx, with thinking enabled.
Question: "Give me a python code snippet that opens a file and sorts the lines of text. "
In 2.4 seconds it gave me 4 examples that work with different sorting configurations and a summary of when to use each.
Compare that to an older model of gpt-oss:20b, took 5 seconds to finish thinking and 2 seconds to stream the answer. It gave me one python example snippet and two one liners that do the same thing.
We are talking about models of the flash size, 100s of billions of parameters, don't listen to the media, size does matter
I was just pointing out your claim that you can't get a response in 3 seconds. If I had asked the model just for the code it was under a second.
Local models are good enough that it's not an issue.
But keep changing the goalposts if it makes you happy.