Comment by somethingsome

Thanks, I just tried, for me it worked on 2x L40S with vLLM. I had some issues due to the model name, forge was forwarding 'default' instead of the real model name 'Qwen2.5-Coder-14B-Instruct'.

If someone else struggle on this step, I added in vLLM args: --served-model-name "Qwen2.5-Coder-14B-Instruct" --served-model-name "default"

So default becomes an alias.

I didn't yet test Forge, I was just happy that it worked at the moment ;)