Comment by leoqa
6 hours ago
I have 3 3090s, all in various gaming computers. I’m not sure what to do with them- the VRAM is only 24GB so doesn’t seem worth the power costs to run an open weight model. Any suggestions on how I can run them as some type of cluster? Is it worth exploring NVLink?
As others said, you can run more than one at a time, but even singly, 24 GB can be very useful in a hybrid approach. For instance, I recently had to process hundreds of high-res blueprints. Used Fable for the main orchestration, but all of the OCR and other work I did with a small local model. Didn't waste expensive Fable tokens on simple grunt work.
You can layer models across multiple GPUs for inference. If you pooled it into one system, that would be 72gb of some pretty damn fast memory. Would be great for dense models in the 30-50b range.
Club 3090 will make installing llm easier, for 1 or more cards.
You can put dual 3090's into one system.
https://www.reddit.com/r/LocalLLaMA/comments/1m5fkts/my_prac...
[dead]