Comment by tarruda
10 hours ago
200B of those 552B is PLE, which works more like a database that is read for each token, thus can be offloaded to a fast SSD.
10 hours ago
200B of those 552B is PLE, which works more like a database that is read for each token, thus can be offloaded to a fast SSD.
Id love an ELI5 for PLE. Im trying to work it into my back of the napikin math for compute vs memory bandwidth limitations on tok/s in PP vs TG work.
My attempt at a simplification of this article on it https://sebastianraschka.com/llm-architecture-gallery/per-la... into a couple of sentences is that they are linear embeddings of the input token space projected per layer, which are then gated by the transformer outputs per layer.
This would mean that the only one set of weights for the ple path needs to be pumped across the memory bandwidth as they are the same linear weights for all layers?
Sheit, maybe im trying to simplify something that i need to look at in detail. but id love to leverage others understanding if possible
While he avoids using the actual PLE acronym, he does actually describe the concept quite well. I think you may enjoy this video. Specifically around 5 minutes into the video is the part you're looking for.
https://www.youtube.com/watch?v=1--PzaHafAU
[dead]
Unfortunately no, it's 200B + 552B. It's not as bad as it sounds though, because most of 552B is in 4bit natively.
Darn, for a moment I thought it would be possible to run in 128G T.T
[flagged]