Comment by a-dub 7 hours ago for a given expert, do you have a sense for what the spatiotemporal access pattern looks like? 2 comments a-dub Reply gitpusher42 1 hour ago Yeah, I checked it. One expert is about a 3.36mb block. If a cache miss happens I read whole block with one pread.And there is some reuse. ~41% selected again for the next token, ~57% within two. Each layer has its own experts, so no reuse between these layers.
gitpusher42 1 hour ago Yeah, I checked it. One expert is about a 3.36mb block. If a cache miss happens I read whole block with one pread.And there is some reuse. ~41% selected again for the next token, ~57% within two. Each layer has its own experts, so no reuse between these layers.
Yeah, I checked it. One expert is about a 3.36mb block. If a cache miss happens I read whole block with one pread.
And there is some reuse. ~41% selected again for the next token, ~57% within two. Each layer has its own experts, so no reuse between these layers.