Comment by thomasjb
7 days ago
I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)
Toss the rtx into a cheapo optiplex or thinkcenter, and run it headless - the load on your machine is gonna make doing other stuff while it’s running painful. Plus that frees up the rest of your vram.
Running an llm on a PC running a desktop environment really isn't that bad. You lose a bit of ram & vram but that only matters if you _reaaaally_ want to push to the max model size your hardware can handle.
The biggest issue I've found is absent mindedly opening YouTube or the like that spike ram requirements and freezing the system up. But that's a me problem
Why would they do that when they already have a perfectly good PC?
…? I said why.
> the load on your machine is gonna make doing other stuff while it’s running painful
Is your question about something else?
I aspire to someday move up to an AM5 system, but for now, the Dell T3610 has to do everything. Might get a second GPU for it though.
start saving your money.
Or watch and wait as models get denser