Comment by mdp2021

10 hours ago

How do you use this model locally, similarly to using `llama-server -m <model>`?

(I mean: outside direct or substantial use of Python, and running the Neural Network in the most efficient way.)

31 comments

mdp2021

Reply
peri-cl  7 hours ago

I've set it up on my local machine just now, as my first local image diffuser. I can confirm it's very easy.

I tried stable-diffusion.cpp, following its compile guide here[0], and its Qwen Image-2.1 specific instructions here[1]. It works out of the box. I made a test pelican[2]. It took 3 minutes on a CPU.

[0] https://github.com/leejet/stable-diffusion.cpp/blob/master/d...

[1] https://github.com/leejet/stable-diffusion.cpp/blob/master/d...

[2] https://i.ibb.co/yMknC2K/output.png

  • mdp2021  3 hours ago

    Thank you! Can you please check how much RAM does it consume (and require)?

    • peri-cl  3 hours ago

      This is what the runtime reports, at Q8:

          total params memory size = 15645.19MB (VRAM 15645.19MB, RAM 0.00MB):
    text_encoders 7669.77MB(VRAM),
    diffusion_model 7331.05MB(VRAM),
    vae 644.38MB(VRAM),
    controlnet 0.00MB(N/A),
    extensions 0.00MB(N/A)

      1 reply →

rwmj  8 hours ago

Additional question is what kind of local hardware would be required for this? 7B parameters sounds very light weight, but I'm not sure. (Edit: The download is 33 GB).

Edit x2: As usual I'm in a twisty maze of pip packages that don't work together, with obscure errors about missing modules, even though I followed the instructions on the page to the letter. I really wish people didn't use Python for this stuff. A simple C/C++ program would be so much better.

  • peri-cl  7 hours ago

    It's about 16 GiB at Q8 quants (combining both the image and language parts). (Meaning, community quantized models from HuggingFace).

    I think it will technically run on anything that has enough memory. I just tried it on a standard laptop (dual-channel DDR5), and it took about 3 minutes for a 512x512. If you'd want to run it at interactive speeds, you would want a GPU (one which fits this in VRAM).

    > "I really wish people didn't use Python for this stuff. A simple C/C++ program would be so much better."

    You mean besides stable-diffusion.cpp ?

    • rwmj  6 hours ago

      > You mean besides stable-diffusion.cpp ?

      Yes, thanks, I didn't know about that. Will try it.

Iolaum  9 hours ago

There is difussion.cpp which is intended for those types of models. I set up krea-2-turbo with the help of ChatGPT 2 months ago, if you have a capable computer that's what I would suggest once it becomes supported.

iugtmkbdfil834  2 hours ago

I am on AI max 395, comfyUI+qwen models is all you technically need. With today's release, I just built a quick and dirty html that allow simpler prompt use and edits ( via headless comfyui ).. its not bad for a day's work, but a little too unpolished to publish. I would say, try comfyUI first ( complex, but it worked OOTB ).

finnjohnsen2  5 hours ago

I use opencode + <a decent saas llm> to set up all this new ai generation stuff. GLM-5.3 is my current gun. Safely inside podman containers too because I dont trust this fast moving python eco system at all. Never do I want this running on my main OS.

I have FLUX.2 klein and dev, Ideogram, LaDA-Image and SenseNova locally. Works great. Ive never touched a file.

The days of making container yamls myself is over. I read them but I dont edit anymore.

fp64  10 hours ago

on the linked GitHub page they list support Diffusers, ComfyUI, vLLM-Omni, SGLang, and LightX2V with links to each

  • mdp2021  9 hours ago

    > Diffusers, ComfyUI, vLLM-Omni, SGLang, and LightX2V

    I think that's all Python (not a direct executable).

    You could just do (see the "Quick Start") four `pip install` and have a dozen lines script to generate the image. But `llama.cpp` and similar do not require e.g. installing Torch (or PyTorch) - you can use `llama.cpp` on a non-specialized machine.

leumon  6 hours ago

Unsloth Desktop is the easiest way imo. There are already gguf quants of this model, or simply wait until the official one comes out.

embedding-shape  10 hours ago

Probably ComfyUI is one of the easiest way to get started with local image/video models. Or perhaps vLLM, if they have support for it already, would be something like `vllm serve <model> --omni --port 9080`