Comment by lelandbatey

I can get 128k context on a 5070ti with 16 GB of VRAM (using the Unsloth 2-bit quant[0]). This is via a .bat file on Windows 11. I'm getting about 50-60 tokens/second and the quality is much higher than Qwen 3.6 27B. I'm using llama.cpp[1]:

    llama-server.exe ^
        -m "Qwen3.8-27B-UD-Q2_K_XL.gguf" ^
        --presence-penalty 0.0 ^
        --repeat-penalty 1.0 ^
        --fit-ctx 128000 ^
        -ctk q4 0 ^
        -ctv q4 0 ^
        --reasoning-budget -1 ^
        --chat-template-kwargs "{\"preserve thinking\": true}" ^
        --host 0.0.0.0 ^
        --port 8033

[0] https://huggingface.co/unsloth/Qwen3.8-27B-GGUF (UD-Q2_K_XL)

[1] https://github.com/ggml-org/llama.cpp/releases

Instructions if you want to do the same:

1. download two files llama-b10434-bin-win-cuda-13.3-x64.zip and cudart-llama-bin-win-cuda-13.3-x64.zip from that llama.cpp Github releases page, and extract both into the same folder.

2. Download the Qwen3.8-27B-UD-Q2_K_XL.gguf file from huggingface and put it into the same folder beside the `llama-server.exe`.

3. Create a file named "RUN_QWEN_3.8.bat" next to `llama-server.exe` and put the text above into that bat file. Double-click the bat file, then open http://localhost:8033 in your browser to see a chat window.

You can use it with any agents by pointing them at http://localhost:8033/v1 which is a working OpenAI compatible endpoint (it doesn't use a token, if you give one it's ignored).

Congratulations, you're now running Qwen 3.8 27B.

Note: I built the computer in question for playing games, yes it needed to be Windows 11 for anticheat reasons to play games with family, I didn't want to dual boot so here I am. I figure I should share instructions for folks who may also have a Windows PC around for such purposes. Specs for this are AMD 9800X3D, 32GB of system RAM, RTX 5070Ti 16GB