Comment by bilekas

2 days ago

I'll actually defend you on this one, but not only the command, that's more of a fine tuning option. On the hugging face link for example, there's no flat 'download' area. It's completely unintuitive for people to know how to use it. Reminds me of my first time trying to make sense of torrents when they first arrived.

There are some clients that will index the models and allow you to do that but I'm no expert, I've used OLama studio but it always seems to go weird for me.

Even this command above, it's not clear where op got the model from. So I'm with yah.

For example, op uses : Qwen3.8-27B-IQ4_NL.gguf.. But I cant see where to download it. It's not tagged on hugging face at least..

12 comments

bilekas

Reply
MrDrMcCoy  2 days ago

Official llama.cpp releases ship with huggingface support. If you don't want to download it yourself, you can just use the `repo/model:quant` convention and it will handle downloading locally for you.

  • bilekas  2 days ago

    But you're assuming I'm using the Olama studio. This model as far as I see doesn't have a gguf download.. Unless I'm missing something on the page.

    If I want to download the model myself, it's not clear. I thought it was supposed to behave like a package manager. But even in nuGet I can download a zip of the package.

    • MrDrMcCoy  2 days ago

      What on earth are you talking about? llama.cpp != Ollama. You can (and should) just use llama.cpp directly. Upstream llama.cpp can take the shorthand huggingface path and automagically download it into a cache folder as part of the launch. Have you read any of the docs?