Comment by itkovian_
6 hours ago
Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.
6 hours ago
Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.
And adding to that, there is also the recurrent state in the Kimi Delta Attention. I wouldn't call it position information but more sth like "position sensitivity"