Comment by gdevenyi
15 hours ago
My 1080ti is my local inference machine now. With the release of the Bonsai 27B models I can run a genuine dense model with usable context. GOAT card.
15 hours ago
My 1080ti is my local inference machine now. With the release of the Bonsai 27B models I can run a genuine dense model with usable context. GOAT card.
What exact runtime are you running Bonsai 27B with right now and what variants of the weights are you using?
I have a 1080ti currently not plugged in. More info please.
Plug it in, get the cuda drivers, run ollama or equivalent, and go from there. I use an 4GB Quadro card from the same generation and depending on the model can get faster than I can read output. It’s not as good as commercial models, but smaller Qwens and Gemmas work well. The Quadro came with a Dell refurb that cost about as much as the card would have by itself at the time.
(I’m doing this with passthrough on a FreeBSD host with bhyve to a Linux guest. Works great.)
I’d like a beefier card, but until prices come down, even older cards like this can do some interesting things if you give them enough time.