← Back to context

Comment by LorenDB

6 months ago

Additionally, Ollama makes model installation a single command. With llama.cpp, you have to download the raw models from Huggingface and handle storage for them yourself.

2 comments

LorenDB

Reply
trissi1996  6 months ago

Not really, llama.cpp can download for quite some time, not as elegant as ollama but:

    llama-server --model-url "https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF/resolve/main/DeepSeek-R1-Distill-Qwen-32B-IQ4_XS.gguf"

Will get you up and running in one single command.

  • yencabulator  6 months ago

    And now you need a server per model? Ollama loads models on-demand, and terminates them after idle, all accessible over the same HTTP API.