Comment by car
12 hours ago
The choice of model could be up to the user, depending on the available RAM on their M class Mac. Newer and bigger usually being smarter.
Llama.app/.cpp and Unsloth Studio on Mac already provide guidance to pick the optimal model for the available memory, all model wheights hosted by huggingface. Both apps can use the local huggingface model cache locally.
If you make it configurable, the user could also run a separate machine with an NVidia card and Linux to serve tokens on the LAN. Or use OpenRouter or any other cloud hosted model offering (DeepSeek, Z.ai, etc.).
I should add that Llama.app and Unsloth Studio provide OpenAI-compatible API endpoints (both use Llama.cpp as their backend).
https://llama.app/docs/api
https://unsloth.ai/docs/new/studio#unsloth-as-an-api-endpoin...