← Back to context Comment by RGS1811 12 hours ago Forgive the ignorance, but how are you running a 763B parameter model on 64GB of RAM? 5 comments RGS1811 Reply liuliu 12 hours ago SSD weights streaming. The actual weights is 552B, then engram. pumanoir 11 hours ago which engine? liuliu 10 hours ago https://releases.drawthings.ai/p/public-beta-of-local-code-b... mococa 12 hours ago By “quantitizing” it ar-nelson 12 hours ago The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.
liuliu 12 hours ago SSD weights streaming. The actual weights is 552B, then engram. pumanoir 11 hours ago which engine? liuliu 10 hours ago https://releases.drawthings.ai/p/public-beta-of-local-code-b...
pumanoir 11 hours ago which engine? liuliu 10 hours ago https://releases.drawthings.ai/p/public-beta-of-local-code-b...
mococa 12 hours ago By “quantitizing” it ar-nelson 12 hours ago The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.
ar-nelson 12 hours ago The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.
SSD weights streaming. The actual weights is 552B, then engram.
which engine?
https://releases.drawthings.ai/p/public-beta-of-local-code-b...
By “quantitizing” it
The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.