Comment by zambelli
21 days ago
Thanks! Did you try it with lmstudio? I actually never tried it with that. Only published ollama, llamfile, llama.cpp native/prompt - and unofficially tested vLLM, but never lmstudio.
21 days ago
Thanks! Did you try it with lmstudio? I actually never tried it with that. Only published ollama, llamfile, llama.cpp native/prompt - and unofficially tested vLLM, but never lmstudio.
Yessir! Been a longtime fan of it, I've spent too many fuckin years wrangling python, especially pytorch, especially on AMD, dep issues for fun and profit... they don't get enough flowers. It's oai compat, no thorns.
I couldn't get it working with lmstudio. I have bonsai-8B running with llama-cpp and am attempting to build a harness for it. Looking good so far, I just got it started but Forge made tool calling work pretty quickly!
Very cool! I'll try to get an issue open on lmstudio support and add it to the backlog.
Maybe you can update the guide and tell us how to use vLLM with Forge when you find some time?
[dead]