Comment by tarruda
15 hours ago
Amazing project.
My only blocker for installing Asahi on my Mac Studio (M1 ultra) is that llama.cpp performance is very bad when compared to using the Metal backend on the same hardware.
15 hours ago
Amazing project.
My only blocker for installing Asahi on my Mac Studio (M1 ultra) is that llama.cpp performance is very bad when compared to using the Metal backend on the same hardware.
Right!? Linux can be configured to idle at less than 500mb of ram - by contrast, with some aggressive shenanigans, MacOS can idle at 2-3gb.
When running inference, I would love to use all of my hardware please Apple.
It idles at 2-3GB but when your apps need the RAM, does it free up system memory so that it's closer to 500MB?
I use my Mac as a headless server to run LLMs, and never even start a desktop session so it always stays at pre-login state without any applications running. Even so, I never saw it using less than 2G in htop when logged via SSH.
I have similar issues with my ultra. Have put it on a laptop, but my desk Setup with my multiple desktop computers is usb-c video based ( kind of a poor man’s kvm) , and asahi doesn’t do usb-c video yet