Comment by remexre

For each token generated, you only send one token’s worth between layers; the previous tokens are in the KV cache.