Plug it in, get the cuda drivers, run ollama or equivalent, and go from there. I use an 4GB Quadro card from the same generation and depending on the model can get faster than I can read output. It’s not as good as commercial models, but smaller Qwens and Gemmas work well. The Quadro came with a Dell refurb that cost about as much as the card would have by itself at the time.
(I’m doing this with passthrough on a FreeBSD host with bhyve to a Linux guest. Works great.)
I’d like a beefier card, but until prices come down, even older cards like this can do some interesting things if you give them enough time.
Plug it in, get the cuda drivers, run ollama or equivalent, and go from there. I use an 4GB Quadro card from the same generation and depending on the model can get faster than I can read output. It’s not as good as commercial models, but smaller Qwens and Gemmas work well. The Quadro came with a Dell refurb that cost about as much as the card would have by itself at the time.
(I’m doing this with passthrough on a FreeBSD host with bhyve to a Linux guest. Works great.)
I’d like a beefier card, but until prices come down, even older cards like this can do some interesting things if you give them enough time.