Comment by wombat23

2 hours ago

I managed to run it with RTX 3070 (8GB VRAM) following the "Quickstart" on HF model card with minor modifications (modify the architecture 86 for your own hardware):

  git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
  cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 && cmake --build build -j

Then downloaded & verified `Ternary-Bonsai-2-27B-PTQ1_0.gguf` from HF and ran

  ./llama.cpp/build/bin/llama-cli -m Ternary-Bonsai-2-27B-PTQ1_0.gguf -ngl 99 -fa on -c 32768 -b 256 -ub 64 --temp 1.0 --top-p 0.95 --top-k 20 -ctk q8_0 -ctv q8_0 -p "hello world in x86 assembler" -n 256

the parameters were suggested by gpt-5.6-luna to reduce memory footprint, as the defaults ran OOM on my gpu. result looks good:

  [ Prompt: 165.6 t/s | Generation: 40.6 t/s ]

would be nice if they upstreamed their changes so that it runs with the original llama.cpp

1 comment

wombat23

Reply