Comment by rahimnathwani

If you want to download the gguf to your regular huggingface cache directory instead of to /tmp, you can download the model and run the server in one step:

  export HF_TOKEN=xxx # optional, speeds up the download
  
  ./llama-prism-b10685-7dffb15/llama serve \
    -hf prism-ml/Ternary-Bonsai-2-27B-gguf:PTQ1_0 \
    --port 8331 -ngl 99 -fa on -c 32768