Comment by JyB
16 hours ago
Why would weights need to be « loaded again » for the 2nd pass? Weights never change at inference time no?
16 hours ago
Why would weights need to be « loaded again » for the 2nd pass? Weights never change at inference time no?
They need to be loaded into shared memory. The weights might fit in global memory if the VRAM is big enough, but they still need to be moved to shared memory for computation.
[flagged]