Comment by monster_truck

21 days ago

Hey this genuinely _fucks_, you're a legend. You can even get stupid good results from the 1 bit bonsai models! Plays v nice with lmstudio

It's now completely reasonable to throw a 7900XTX in a spare rig, put it in the basement, give it an absurd goal, and forget about it.

6 comments

monster_truck

Reply
zambelli  21 days ago

Thanks! Did you try it with lmstudio? I actually never tried it with that. Only published ollama, llamfile, llama.cpp native/prompt - and unofficially tested vLLM, but never lmstudio.

  • monster_truck  20 days ago

    Yessir! Been a longtime fan of it, I've spent too many fuckin years wrangling python, especially pytorch, especially on AMD, dep issues for fun and profit... they don't get enough flowers. It's oai compat, no thorns.

  • tmjdev  21 days ago

    I couldn't get it working with lmstudio. I have bonsai-8B running with llama-cpp and am attempting to build a harness for it. Looking good so far, I just got it started but Forge made tool calling work pretty quickly!

    • zambelli  21 days ago

      Very cool! I'll try to get an issue open on lmstudio support and add it to the backlog.