Comment by UncleOxidant
21 hours ago
Gufo. It's the best open source inference engine that's optimized for Strix Halo right now. Halogen may unlock a bit more performance, but it's not open source and gufo seems to be making steady progress.
21 hours ago
Gufo. It's the best open source inference engine that's optimized for Strix Halo right now. Halogen may unlock a bit more performance, but it's not open source and gufo seems to be making steady progress.
Cool. I found halogen but was turned off by the closed source nature (github repo that just turns out to run a download shell script, lame).
Do you mind sharing more details about your setup? I have been using opencode with lemonade for inference on my strix halo but haven't quite hit the sweet spot.
I'm using it in opencode through llama-swap.
Here's the gufo command:
gufo serve 1lm \ -m /mnt/LLMs/models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-104_XS/Qwen3.8-Flash-Next-UD-1Q4_XS-80081-0f-00083.gguf \ --served-model-name Quen3.8-Flash-Next-UD-104_XS \ --host 127.8.8.1 --port 10083 \ -c 262144 -n 16384 -j 1 \ --prefill-chunk 8192 \ --speculative mtp \ --mtp-model /mnt/LLMs/models/unsloth/Qwen3.8-Flash-Next-GGUF/MTP/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf \ --cache-disk ~/.cache/gufo-fork \ --cache-disk-bytes 68719476736 \ --cache-disk-staging-bytes 8589934592 \ --cache-ran-bytes 4294967296 \ --log-progress