← Back to context Comment by RGS1811 8 hours ago Forgive the ignorance, but how are you running a 763B parameter model on 64GB of RAM? 5 comments RGS1811 Reply liuliu 8 hours ago SSD weights streaming. The actual weights is 552B, then engram. pumanoir 7 hours ago which engine? liuliu 6 hours ago https://releases.drawthings.ai/p/public-beta-of-local-code-b... mococa 8 hours ago By “quantitizing” it ar-nelson 8 hours ago The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.
liuliu 8 hours ago SSD weights streaming. The actual weights is 552B, then engram. pumanoir 7 hours ago which engine? liuliu 6 hours ago https://releases.drawthings.ai/p/public-beta-of-local-code-b...
pumanoir 7 hours ago which engine? liuliu 6 hours ago https://releases.drawthings.ai/p/public-beta-of-local-code-b...
mococa 8 hours ago By “quantitizing” it ar-nelson 8 hours ago The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.
ar-nelson 8 hours ago The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.
SSD weights streaming. The actual weights is 552B, then engram.
which engine?
https://releases.drawthings.ai/p/public-beta-of-local-code-b...
By “quantitizing” it
The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.