Comment by RGS1811 9 hours ago Forgive the ignorance, but how are you running a 763B parameter model on 64GB of RAM? 5 comments RGS1811 Reply liuliu 9 hours ago SSD weights streaming. The actual weights is 552B, then engram. pumanoir 8 hours ago which engine? liuliu 7 hours ago https://releases.drawthings.ai/p/public-beta-of-local-code-b... mococa 9 hours ago By “quantitizing” it ar-nelson 9 hours ago The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.
liuliu 9 hours ago SSD weights streaming. The actual weights is 552B, then engram. pumanoir 8 hours ago which engine? liuliu 7 hours ago https://releases.drawthings.ai/p/public-beta-of-local-code-b...
pumanoir 8 hours ago which engine? liuliu 7 hours ago https://releases.drawthings.ai/p/public-beta-of-local-code-b...
mococa 9 hours ago By “quantitizing” it ar-nelson 9 hours ago The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.
ar-nelson 9 hours ago The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.
SSD weights streaming. The actual weights is 552B, then engram.
which engine?
https://releases.drawthings.ai/p/public-beta-of-local-code-b...
By “quantitizing” it
The smallest quant is still over 100GB, so I don't know how that fits in 64GB RAM. It would have to be partly streamed from disk.